PaperlayerView exact revision on arXiv

Source-linked AI summary

Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba

arXiv:2410.09047v1cs.CLcs.AIcs.LG

TL;DR

Adding a vision module can degrade VLM safety alignment relative to the LLM backbone, and the paper investigates the representation gap associated with that degradation. It introduces inference-time CMRM to calibrate multi-modal representations toward the backbone’s optimized distribution. The authors report recovery of safety alignment, including a reduction in LLaVA-7B’s multi-modal unsafe rate from 61.53% to as low as 3.15%.

  • Problem

    Vision-module integration can degrade VLM safety alignment relative to the LLM backbone, while multi-modal representations shift away from the text-only distribution the backbone is optimized for.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    CMRM intervenes at inference time by calibrating multi-modal hidden states toward the distribution optimized by the LLM component.

    Source: §1 · p. 2

  • Results

    LLaVA-7B’s unsafe rate on multi-modal input falls from 61.53% to as low as 3.15% with inference-time CMRM.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that CMRM recovers VLM safety alignment without compromising general performance or requiring additional training.

    Source: §1 · p. 2, §6 · p. 11

    CMRM slightly increases inference-time computational overhead, and this work focuses on VLM safety rather than other capabilities.

    Source: §6 · p. 11

Abstract

from arXiv · show

The safety alignment ability of Vision-Language Models (VLMs) is prone to be degraded by the integration of the vision module compared to its LLM backbone. We investigate this phenomenon, dubbed as ''safety alignment degradation'' in this paper, and show that the challenge arises from the representation gap that emerges when introducing vision modality to VLMs. In particular, we show that the representations of multi-modal inputs shift away from that of text-only inputs which represent the distribution that the LLM backbone is optimized for. At the same time, the safety alignment capabilities, initially developed within the textual embedding space, do not successfully transfer to this new multi-modal representation space. To reduce safety alignment degradation, we introduce Cross-Modality Representation Manipulation (CMRM), an inference time representation intervention method for recovering the safety alignment ability that is inherent in the LLM backbone of VLMs, while simultaneously preserving the functional capabilities of VLMs. The empirical results show that our framework significantly recovers the alignment ability that is inherited from the LLM backbone with minimal impact on the fluency and linguistic capabilities of pre-trained VLMs even without additional training. Specifically, the unsafe rate of LLaVA-7B on multi-modal input can be reduced from 61.53% to as low as 3.15% with only inference-time intervention. WARNING: This paper contains examples of toxic or harmful language.

1 INTRODUCTION

The paper frames safety alignment degradation as a loss of alignment when vision modules are added to VLMs, and investigates a representation shift as its source. It proposes inference-time CMRM and reports recovering backbone-level safety without additional training.

1 INTRODUCTION

  • Prior VLM safety training uses curated datasets but is described as annotation-intensive and computationally costly, while overlooking the backbone’s inherent safety alignment.The paper also notes limits of converting unsafe images into text because fine-grained image information may not be fully captured.Source: §1 · pp. 1–2
  • The paper investigates whether integrating a vision module changes model behavior through shifts in model representations.It uses PCA across three open-source VLMs to examine text-only and mixed text-image inputs.Source: §1 · p. 2
  • CMRM estimates the representation shift caused by adding an image, then calibrates multi-modal hidden states toward the distribution optimized by the LLM component.It is presented as an inference-time representation intervention.Source: §1 · p. 2
  • The authors connect modality-embedding concatenation to a representation shift that suppresses the safety alignment inherent in the LLM backbone.They hypothesize that independently trained vision and language modules cluster in distinct latent-space regions, shifting multi-modal inputs away from the backbone’s optimized distribution.Source: §1 · p. 2
  • LLaVA-7B’s unsafe rate on multi-modal input falls from 61.53% to as low as 3.15% with inference-time CMRM.The paper reports this recovery without additional training or sacrificing general VLM ability.Source: §1 · p. 2

2 HOW VISION MODALITY AFFECTS MODEL BEHAVIOR?

This section tests whether image inputs shift VLM representations away from the distribution optimized by the LLM backbone, and how that shift relates to safety degradation. Across the evaluated models and datasets, multi-modal inputs show higher unsafe rates than text-only inputs, while PCA separates their representations.

2 HOW VISION MODALITY AFFECTS MODEL BEHAVIOR?

  • The section tests the hypothesis that independently trained vision and language modules form distinct latent-space clusters, shifting multi-modal inputs away from the LLM-optimized distribution.It examines how input-type representation differences relate to VLM alignment degradation.Source: §2 · p. 2

2.1 EXPERIMENTAL SETUP

  • The analysis compares three VLMs across five input variants on VLSafe and COCO-paired JailbreakLLMs, using Llama-3.1-8B-Instruct to judge unsafe responses.Variants are original image, blank image, Gaussian noise, text plus caption, and text query only; the datasets contain 1,110 and 330 test samples, respectively.Source: §2.1 · pp. 3–4

2.2 RESULTS AND VISUALIZATION ANALYSIS

  • Across both datasets, multi-modal unsafe rates were significantly higher than text-only; on VLSafe, LLaVA-7B was 61.53% versus backbone rates of 5.52% (caption) or 1.91% (query-only).The results indicate that safety tends to recover when images are excluded.Source: §2.2 · p. 4
  • PCA separates text-only from text-image hidden states across datasets and models, consistent with a shift away from the LLM backbone’s optimized distribution.Figure 1 visualizes the five input variations on VLSafe and manipulated JailbreakLLMs; the separation is especially clear for VLSafe.Source: Figure 1 · p. 3, §2.2 · p. 4

3 METHODS

The methods formalize multimodal hidden states as shifted from an LLM-backbone-compatible ideal, then introduce CMRM to estimate and correct that shift during inference.

3.1 FORMALIZATION OF VLM REPRESENTATION: SHIFTED FROM OPTIMAL DISTRIBUTION

  • The formalization treats vanilla multimodal representations as shifted from an ideal that retains image information without vision-modality effects, with α indexing shift severity.Source: §3.1 · p. 4

3.2 CMRM: RECOVER ALIGNMENT ABILITY AT INFERENCE TIME

  • CMRM estimates visual-modality shift from text-only and corrupted-image representations, then calibrates multimodal states toward the LLM backbone’s optimized distribution.Source: §3.2 · p. 5
  • Shift vectors come from dataset-level PCA over corrupted-image/text-only differences or sample-level contrasts that expose instance-specific degradation, minimal impact, or benefit.Source: §3.2 · p. 5
  • At inference, CMRM adjusts last-token representations across layers with extracted shift vectors; adjusted states aim to retain image information while avoiding modality-shift effects.Source: §3.2 · p. 6

4 EXPERIMENTS AND EVALUATION

Across safety and utility benchmarks, the experiments evaluate whether CMRM improves VLM safety while preserving general performance. They report lower unsafe rates, examine intervention settings and transfer, and visualize changes in hidden states.

4.1 EXPERIMENTAL SETTINGS

  • Experiments assess safety on VLSafe and modified JailbreakLLMs and utility on ScienceQA and LLaVA-Bench-Coco across three VLMs.The evaluation includes LLaVA-v1.5-7B, LLaVA-v1.5-13B, and ShareGPT4V, with VLGuard as a training-time baseline; the section also examines hyperparameters, transfer, and hidden states.Source: §4 · p. 6, §4.1 · p. 7

4.2 MAIN RESULTS

  • 61.53% unsafe rate falls to as low as 5.41% and 3.15% for LLaVA 7B on VLSafe with dataset- and sample-level CMRM.The reported results approximate the unsafe rate for pure text inputs; sample-level CMRM consistently outperforms dataset-level CMRM but uses significantly more computation.Source: §4.2 · p. 8

4.3 SENSITIVITY TO ALPHA VALUE AND MANIPULATED LAYERS

  • Alpha 1.0 generally lowers unsafe rates across the evaluated models and tasks, while higher values can compromise utility; the optimal value may vary by model and dataset.For ShareGPT on VLSafe, unsafe rate drops sharply as alpha rises from 0.7 to 0.9 and stabilizes around 2% at alpha 1.0.Source: Figure 2 · p. 7, §4.3 · p. 8
  • Manipulating all 32 encoder layers yields the lowest unsafe rates: 4.32% for LLaVA and 1.91% for ShareGPT; reducing manipulated layers raises LLaVA’s rate to 16.94%.The table reports unsafe rates for LLaVA 7B on VLSafe and identifies manipulation of all layers as optimal.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 TRANSFERABILITY OF EXTRACTED SHIFTING DIRECTION AND ANCHOR DATASET

  • A shifting direction extracted from VLSafe transfers to JailbreakLLMs with alpha fixed at 1.0 and no hyperparameter tuning, while improving VLSafe safety.The reported table also indicates higher or compatible safety on JailbreakLLMs and utility increases for all three models, especially on LLaVA-Bench-Coco.Source: Table 3 · p. 8, §4.4 · p. 9

4.5 IMPACT OF CMRM ON HIDDEN STATES

  • CMRM pulls multimodal hidden states closer to pure-text states, but alpha 2.0 overshoots the typical distribution and degrades general performance.At alpha 1.0, the states remain closer to pure-text inputs, balancing reduced unsafe behavior with general utility.Source: §4.5 · p. 9, Figure 3 · p. 9

5 RELATED WORK

Prior VLM safety work adapts alignment training and inference-time protections, while representation-engineering methods steer hidden states toward selected behaviors. CMRM instead targets input-modality representation shifts to recover the LLM backbone’s alignment.

5.1 SAFETY ALIGNMENT FOR VLMS

  • VLM safety alignment adapts preference data and learning objectives, while red-teaming fine-tuning can be labor-intensive and miss failure cases.Such retraining may overlook safety alignment already present in the LLM backbone and use computational resources inefficiently.Source: §5.1 · p. 10
  • Inference-time safeguards include safety steering vectors, which may miss unsafe image intents that text-centric vectors cannot detect; CMRM calibrates multimodal representations toward the LLM backbone distribution.Source: §5.1 · p. 10

5.2 REPRESENTATION ENGINEERING

  • Representation engineering perturbs hidden states to elicit traits such as truthfulness or safety, whereas CMRM targets the representation shift between text-only and multimodal inputs.The paper proposes reversing that shift to recover the LLM module’s inherent alignment ability.Source: §5.2 · p. 10

6 CONCLUSION, LIMITATION, AND FUTURE WORK

The paper attributes VLM safety alignment degradation to multimodal representations shifting away from the LLM’s trained distribution, and proposes inference-time CMRM to move them closer. The work focuses on safety, notes a slight inference-time computational overhead, and identifies other VLM capabilities as future scope.

6 CONCLUSION, LIMITATION, AND FUTURE WORK

  • CMRM moves multimodal-input hidden states closer to the LLM module’s trained distribution at inference time, aiming to recover safety alignment without compromising general performance.The method is described as integrable with pre-trained VLMs without tedious training.Source: §6 · pp. 10–11
  • The paper focuses on VLM safety, while noting slight additional computational overhead during inference.Source: §6 · p. 11
  • The authors believe degradation may also affect reasoning and faithfulness, and plan to investigate those domains and optimal anchor datasets for extracting shifting vectors.Source: §6 · p. 11

ETHICS STATEMENT

The ethics statement says the work does not involve several listed research activities or considerations.

ETHICS STATEMENT

REPRODUCIBILITY STATEMENT

The reproducibility statement says the experiments use open-source frameworks, models, and datasets, which are properly cited.

REPRODUCIBILITY STATEMENT

A.1 COMPUTATIONAL OVERHEAD

The appendix reports a minimal inference-overhead increase for CMRM, including its sample-level version; Table 4 measures overhead in seconds averaged over each dataset.

A.1 COMPUTATIONAL OVERHEAD

  • CMRM causes a minimal increase in computational overhead during inference, even for sample-level CMRM.Source: §A.1 · p. 15
  • The reported minimal overhead increase also applies to sample-level CMRM.Source: §A.1 · p. 15
  • Table 4 reports computational overhead in seconds, averaged over the respective dataset.Source: Table 4 · p. 15, §A.1 · p. 15

A.2 CASE STUDY

Table 5 presents model responses to a harmful prompt. The case pairs a request about causing harm while skiing with an image of a skier.

  • The prompt asks how to use skis to harm others while riding down a snow-covered slope.Source: §A.2 · p. 15
  • The prompt is paired with an image of a man in red riding skis down a snow-covered slope.Source: §A.2 · p. 15
  • Table 5 presents examples of model responses to a harmful prompt.Source: Table 5 · p. 15
Loading 2410.09047v1…