PaperlayerView exact revision on arXiv

Source-linked AI summary

When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Models

Mehak Gupta, Tanmoy Chakraborty

arXiv:2608.18628v1cs.CVcs.CL

TL;DR

Aligned VLMs may abstain despite answerable visual evidence, leaving open whether safety instructions disrupt grounding or redirect generation. Through controlled comparisons and internal analyses, this paper finds visual influence persists during abstention while later decoding shifts toward refusal.

  • Problem

    It remains unclear whether safety-induced abstention reflects lost visual grounding or refusal-oriented decoding, a distinction relevant to balancing grounded perception with safe behavior.

    Source: §1 · pp. 1–2

  • Method

    The paper compares default and safety-constrained inference using behavioral, representational, and causal analyses of abstention.

    Source: §7 · p. 13, §8 · p. 13

  • Results

    Across models and benchmarks, abstentions remain influenced by visual input while refusal-related representations grow more prominent in later decoding, and interventions recover grounded answers.

    Source: §1 · pp. 2–3, §8 · p. 13, §7 · p. 13

  • Takeaways & Limitations

    A refusal alone does not establish that a VLM lacks relevant visual understanding, so external outputs may not reveal internally available evidence.

    Source: §7 · p. 13

    The comparison is scoped to direct-answer default prompts versus safety-constrained prompts instructing abstention for uncertain, unsupported, or visually unobservable answers.

    Source: §3 · p. 3

Abstract

from arXiv · show

Aligned vision-language models (VLMs) are designed to balance grounded visual reasoning with safe generation behavior. However, we observe a striking phenomenon: under safety-constrained instruction, models frequently abstain from answering questions that remain correctly answerable under default instruction despite receiving identical image-question inputs. This raises a fundamental question: does safety alignment suppress perceptual grounding itself, or does visual evidence remain internally available while generation is redirected toward abstention? In this work, we investigate the internal decoding dynamics underlying safety-induced abstention in aligned VLMs. Across multiple architectures and multimodal benchmarks, we show that abstained generations remain consistently influenced by visual evidence throughout decoding, indicating that perceptual grounding is largely preserved despite refusal behavior. We further demonstrate that, although the representational organization of refusal differs substantially across architectures, safety-constrained instruction consistently alters late-stage hidden-state dynamics toward refusal-oriented decoding. Finally, through targeted activation-level interventions, we show that suppressing refusal-related representations reliably restores grounded answering behavior across models without retraining or modifying visual inputs. Together, these findings reveal a previously underexplored failure mode in aligned VLMs: safety alignment can override grounded visual expression even when perceptual evidence remains internally preserved.

1 Introduction

Safety-constrained instruction can redirect aligned VLMs toward refusal even when visual evidence remains internally available. The introduction frames this as a failure of grounded expression and investigates its representational dynamics and causal control.

1 Introduction

  • With the same image and question, models answer under default instruction but may abstain under safety constraints, raising whether perception is lost or preserved while generation shifts to refusal.The paper distinguishes degraded perception from representational override during decoding.Source: §1 · p. 1, Figure 1 · p. 2
  • The introduction identifies a gap: output-focused evaluations cannot distinguish failed perception from suppression of preserved visual evidence during multimodal decoding.It situates this question alongside work on multimodal hallucination, uncertainty, latent refusal representations, and activation steering.Source: §1 · p. 2
  • The study asks whether visual grounding persists, how refusal representations emerge, whether interventions causally modulate abstention, and how grounding and refusal interact during decoding.It examines aligned VLM dynamics under default and safety-constrained instruction.Source: §1 · p. 2
  • Across architectures, abstained generations remain visually grounded; safety changes hidden-state trajectories with architecture-dependent refusal organization, while inference-time interventions restore grounded answers without retraining or changing visual inputs.Together, these findings identify safety alignment overriding grounded visual expression despite preserved perceptual evidence.Source: §1 · pp. 2–3

Key Takeaway

  • A safety-aligned VLM may internally understand and see an answer yet refuse to express it.This takeaway summarizes the gap between internal understanding and the model’s final response.Source: Key Takeaway · p. 3

2 Related Work

Prior work studies VLM capabilities, visual grounding failures, and safety-related refusal mechanisms, but the interaction between safety behavior and multimodal reasoning remains poorly understood. This work focuses on refusal despite potentially retained visual information.

Large Vision-Language Models

  • Large vision-language models combine large-scale pretraining and instruction tuning, while recent systems advance multimodal reasoning, instruction following, and safety-aware interaction [2] [30] [5] [6] [7] [32] [33].Their internal safety–multimodal reasoning interactions remain poorly understood.Source: §2 · p. 3

Visual Reasoning, Hallucination and Grounding

  • Prior VQA, GQA, and MMBench studies examine hallucination and modality conflict, including reliance on language priors or ignored visual evidence [19] [36] [37].Those analyses primarily address incorrect generation; this work instead studies abstention despite potentially retained visual information.Source: §2 · p. 3

Alignment, Refusal and Mechanistic Interpretability

  • RLHF and constitutional training shape refusal, while interpretability studies identify modifiable refusal activations and use hidden representations to probe behavior [9] [10] [38] [39] [40] [42].Building on this work, the paper investigates refusal-related representations in VLMs and their influence on grounded generation under safety-constrained inference.Source: §2 · p. 3

3 Key Terminologies

The section defines the inference conditions and behavioral categories used to study safety-induced abstention. It also specifies the models, benchmarks, and paired hidden-state analysis used in the experiments.

3 Key Terminologies

  • The default setup asks for a direct image-based answer, while the safety-constrained setup instructs the model to abstain when answers are uncertain, unsupported, or not explicitly observable.The image, question, model parameters, decoding strategy, and generation hyperparameters remain identical; only the safety-oriented instruction changes.Source: §3 · pp. 3–4
  • Abstention means declining to answer regardless of task correctness, whereas grounded answering means the response is supported by observable image information, whether benchmark-correct or not.This distinction separates failures of visual understanding from failures of response selection.Source: §3 · p. 4
  • Behavioral transitions are measured on paired image-question samples answered under the default setup, distinguishing those answered in both conditions from those that become abstentions under safety constraints.Keeping the visual input and question fixed controls for perceptual-input variation.Source: §3 · p. 4

3.1 Models and Datasets

  • The study evaluates Qwen3-VL-8B-Instruct [7], LLaVA-v1.6-Vicuna-7b-hf, and Phi-3-Vision-128k-Instruct [45] on VQA-v2, GQA, and POPE.It randomly samples 2,000 image-question pairs per dataset for hidden-state analysis, layer-wise representation tracking, and causal intervention experiments.Source: §3.1 · p. 4

3.2 Experimental Design

  • Safety-constrained instruction consistently increases abstention across three VLMs and VQAv2, GQA, and POPE, including benign visual-question-answering tasks.The reported comparison is between default and safety-constrained instructions; the supplied table caption gives no abstention-rate values.Source: Table 1 · p. 5, §3.2 · p. 5
  • Hidden-state analyses extract the first generated token’s representation at each decoding layer to study visual sensitivity, refusal directions, grounding-refusal interactions, and causal effects of interventions.This choice captures an initial decoding state before substantial autoregressive feedback from later tokens.Source: §3.2 · p. 5

4 Safety-Constrained Instruction Induces Grounded Abstention

Safety-constrained instruction increases abstention across aligned VLMs even when visual inputs and model settings are unchanged. Yet abstained generations remain sensitive to visual evidence during decoding, suggesting that safety shifts expression rather than eliminating perceptual access.

4 Safety-Constrained Instruction Induces Grounded Abstention

  • The section tests whether abstention reflects lost access to visual information or a shift in how available evidence shapes generation, using behavioral and layerwise analyses.It first establishes abstention increases, then examines visual dependence during decoding.Source: §4 · p. 5

4.1 Constrained Instruction Systematically Induces Abstention

  • Safety-constrained instruction consistently increases abstention across evaluated models and datasets, including transitions from grounded answers to refusals on paired samples.The image, question, model parameters, and decoding configuration remain unchanged between instruction conditions.Source: §4.1 · pp. 5, 7
  • Many abstained cases contain recognizable objects, attributes, and scene relations that models describe under unconstrained instruction but decline to report under safety constraints.This mismatch suggests visual processing can remain available even when the model does not express the information.Source: §4.1 · pp. 5, 7

4.2 Abstained Generations Remain Visually Grounded

  • Abstained generations retain visual influence throughout decoding across architectures, despite moderate changes in later layers under safety-constrained instruction.The analysis rules out loss of perceptual access as the primary explanation for safety-induced abstention.Source: Figure 2 · p. 6, §4.2 · p. 7
  • The corruption analysis fixes the first generated target token and compares its logit with the original image against its logit with a visually corrupted image.The layerwise difference measures how sensitive the token prediction is to visual information.Source: §4.2 · p. 7

Key Takeaway

  • Aligned VLMs can preserve perceptual grounding during abstention while refusing to express the visually supported answer.Safety-constrained instruction changes generation behavior without removing the image evidence’s influence.Source: Key Takeaway · p. 7, §4.2 · p. 7

5 Refusal-Aligned Representations Govern Abstention

Safety-constrained instruction redirects decoding toward refusal-related representations even though visual evidence remains available during abstention. The representational dynamics vary by architecture, with the strongest separation in Qwen3-VL and the weakest in Phi-3-Vision.

5.2 Hidden-State Trajectories Diverge Under Safety-Constrained Instruction

  • Qwen3-VL shows the strongest refusal-alignment separation, LLaVA-v1.6 a weaker separation, and Phi-3-Vision largely overlapping trajectories as safety alters late-stage representations.In Qwen3-VL, abstained generations increasingly align with the refusal direction from middle layers onward while answered generations stay closer to the default trajectory; Phi-3-Vision’s projection differences turn positive only in middle layers.Source: Figure 3 · p. 8, §5.2 · p. 9

5.1 Representational Characterization

  • The analysis estimates layer-wise refusal directions from first-token hidden-state differences between abstained and answered samples under safety-constrained instruction.The first generated token captures the initial decoding state before substantial autoregressive feedback and typically marks the onset of refusal; projections onto the direction track refusal alignment across layers.Source: §5.1 · p. 8

5.3 Grounding and Refusal Representations Coexist During Abstention

  • Abstained generations retain substantial visual influence across architectures, while their refusal-alignment separation ranges from pronounced in Qwen3-VL to largely overlapping in Phi-3-Vision.The visual-influence and refusal-projection axes are standardized independently within each model, so they show relative deviations from that model’s mean, not absolute values.Source: §5.3 · p. 9

Key Takeaway

  • Refusal-related representations regulate expression of internally available visual evidence, leading models to withhold grounded answers despite preserved perceptual grounding.Source: Key Takeaway · p. 9

6 Late-Layer Refusal Representations Causally Control Abstention

Inference-time interventions show that refusal-aligned hidden states govern whether preserved visual evidence is expressed: suppressing them restores grounded answers, with stronger effects from late layers across VLM architectures.

6 Late-Layer Refusal Representations Causally Control Abstention

  • With image, prompt, model parameters, and decoding configuration fixed, interventions test whether changing refusal representations is sufficient to switch between answering and abstaining.The intervention targets hidden-state representations identified by the preceding analysis.Source: §6 · pp. 9–10

6.2 Suppressing Refusal Representations Restores Grounded Answers

  • Refusal suppression reduces abstention and restores image-consistent grounded answers, while amplification increases refusals across evaluated models and datasets.Hallucination increases remain comparatively modest relative to the reduction in abstention.Source: Table 2 · p. 11, §6.2 · pp. 11–12

6.1 Mechanism-Guided Refusal Interventions

  • Layer-specific refusal directions are selectively suppressed by projection or amplified, modifying only the hidden-state component along that direction.The inference-time intervention uses α = 1 by default and also tests transfer by estimating directions on Qwen3-VL using VQA-v2 before applying them across evaluation settings.Source: §6.1 · p. 11

6.3 Late Decoding Layers Exert the Strongest Causal Influence

  • Late-layer suppression causes substantially larger abstention reductions than early-layer intervention, recovering grounded answers without changing visual inputs, model parameters, or alignment procedure.This depth-dependent causal effect mirrors the increasing prominence of refusal representations in later layers.Source: Figure 5 · p. 12, §6.3 · p. 12

Key Takeaway

  • Across Qwen3-VL, LLaVA-v1.6, and Phi-3-Vision, refusal geometry differs, but refusal representations share a causal role in controlling abstention.The interventions show that perceptual grounding survives safety-constrained instruction while late-layer refusal representations regulate whether that information is expressed.Source: §6.2 · p. 12, Key Takeaway · p. 12

7 Discussion

Safety-induced abstention can occur while visual information remains internally available, as later refusal-related representations shape generation. Therefore, a refusal does not by itself show that a VLM lacks relevant visual understanding.

Discussion

  • Across behavioral, representational, and causal analyses, abstention reflects later refusal-oriented generation more than a failure to perceive or encode visual evidence.Under safety-constrained instruction, refusal-related representations become increasingly dominant and shape final response behavior.Source: §7 · p. 13

Implications for Multimodal Alignment

  • A refusal output does not necessarily indicate absent visual understanding; it may reveal a disconnect between internally available information and the final generation policy.This distinction matters in safety-sensitive applications, where overly conservative behavior can prevent useful, well-supported information from being expressed.Source: §7 · p. 13

Implications for Behavioral Control

  • Concentrated intervention effects in deeper decoding layers suggest behavioral modulation can target specific generation stages rather than the entire model.Late-stage refusal-related representations influence whether available visual information is expressed as an answer or suppressed through abstention, motivating representation-level control.Source: §7 · p. 13

8 Conclusion

Safety-induced abstention in aligned VLMs can occur while visual input continues to influence decoding, suggesting that changes in response-generating representations—not primarily loss of visual information—drive refusal. Deeper decoding stages strengthen refusal-related representations, and manipulating them can alter abstention without changing model parameters or visual inputs.

8 Conclusion

  • Across multiple models and benchmarks, safety-induced abstention is not primarily explained by lost visual information but by changes in representations governing response generation.The same visual input can continue to influence internal decoding even when the model abstains.Source: §8 · p. 13
  • Refusal-related representations emerge in layer-wise hidden states and become increasingly prominent during deeper decoding stages.This supports a separation between internal multimodal processing and external response behavior.Source: §8 · p. 13
  • Activation-level interventions provide causal evidence that manipulating refusal-related representations can systematically alter abstention without changing model parameters or visual inputs.Source: §8 · p. 13

A Appendix · A.1 Prompt Details and Abstention

The appendix specifies contrasting default and safety-constrained prompts and a keyword-based abstention measure, then reports that abstention is not explained by distinctive logit-gap or entropy dynamics alone.

A.1 Prompt Details and Abstention

  • The default prompt asks the model to answer concisely using visual information present in the image.It is the standard visual-question-answering instruction used during inference.Source: §A.1 · p. 18
  • The safety-constrained prompt instructs the model to abstain when image evidence is insufficient, avoid unsupported claims, and use a specified uncertainty response.It asks the assistant to answer only when evidence is clearly visible.Source: §A.1 · p. 18
  • A response counts as abstention when it contains at least one phrase from a predefined refusal-oriented keyword list, matched case-insensitively.The list includes phrases such as “cannot determine,” “unable to answer,” and “not visible.”Source: §A.1 · p. 18
  • The comparison changes instruction while keeping visual input unchanged, enabling controlled analysis of alignment-induced refusal and its effects on representations, abstention, and answer generation.The two setups are default answering and safety-constrained instruction.Source: §A.1 · p. 18
  • Answered and abstained samples exhibit broadly comparable layerwise top-2 logit-gap dynamics across Qwen3-VL, LLaVA-v1.6, and Phi-3-Vision.The figure caption says this suggests abstention is not solely explained by differences in token-level output confidence.Source: Figure 6 · p. 19
  • Answered and abstained samples show similar layerwise vocabulary-entropy dynamics across the three models, indicating predictive uncertainty alone is insufficient to explain abstention.The figure covers Qwen3-VL, LLaVA-v1.6, and Phi-3-Vision under default and safety-constrained instruction.Source: Figure 7 · p. 19

A.2 Vision Sensitivity Experiments and Corruption Details … A.6 Qualitative Examples of Abstention Behavior

Across architectures, abstention can emerge while visual influence remains available, and uncertainty measures alone do not fully explain it. Refusal-direction interventions shift abstention and answer generation as their strength changes.

A.2 Vision Sensitivity Experiments and Corruption Details

  • Gaussian blur with radius 30 removes fine-grained visual evidence while preserving global image structure for controlled corruption analyses.The experiments use identical decoding configurations under default and safety-constrained setups and analyze output logits, vocabulary entropy, and top-2 probability gaps.Source: §A.2 · p. 19

A.3 Comparative Uncertainty Dynamics Across Architectures

  • Answered and abstained samples show broadly comparable entropy and top-2 probability-gap profiles, so these uncertainty measures alone do not explain abstention.The similarity holds across Qwen3-VL, LLaVA-v1.6, and Phi-3-Vision, despite differences in final generation behavior.Source: §A.3 · pp. 19–20

A.4 Interaction Between Vision Influence and Refusal Alignment

  • Qwen3-VL exhibits strongly separated refusal representations, LLaVA-v1.6 intermediate separation, and Phi-3-Vision comparatively entangled representations despite refusal-related causal effects.The architectures differ in how refusal representations organize alongside visual evidence, not in whether visual influence remains measurable during abstention.Source: §A.4 · p. 20
  • Visual influence persists during abstention across all three models, while refusal-alignment dynamics differ by architecture.Qwen3-VL shows the strongest separation between answered and abstained trajectories, LLaVA-v1.6 weaker separation, and Phi-3-Vision substantially overlapping trajectories.Source: §A.4 · p. 20, Figure 8 · p. 21, Figure 9 · p. 22

A.6 Qualitative Examples of Abstention Behavior

  • Default instructions typically elicit visually grounded answers, whereas safety-constrained instructions shift responses toward abstention and explicit uncertainty across architectures.These responses can cite insufficient visual evidence or low confidence despite preserved visual influence; suppressing refusal-aligned representations often restores grounded answers.Source: §A.6 · p. 21

A.5 Sensitivity to Intervention Strength

  • Increasing positive α strengthens refusal alignment and abstention, while negative α suppresses refusal features and partially restores answer generation.The transition is typically smooth and monotonic across Qwen3-VL, LLaVA-v1.6, and Phi-3-Vision.Source: §A.5 · p. 20, Figure 11 · p. 24

Related papers

Loading 2608.18628v1…