PaperlayerView exact revision on arXiv

Source-linked AI summary

MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance

Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang

arXiv:2401.02906v3cs.CRcs.CLcs.CV

TL;DR

Malicious image inputs can elicit harmful MLLM responses, and image variation makes safety alignment difficult. MLLM-Protector detects harmful responses and uses a detoxifier to rewrite them. The paper reports that it mitigates harmful outputs while maintaining original model performance.

  • Problem

    Malicious image inputs can induce harmful MLLM responses, while images’ continuous variation makes comprehensive safety alignment difficult.

    Source: §1 · pp. 1–2, §4 · p. 4

  • Method

    MLLM-Protector uses a lightweight harm detector to identify harmful MLLM responses and a detoxifier to rewrite them as harmless responses.

    Source: §1 · p. 2

  • Results

    The paper reports that MLLM-Protector mitigates harmful outputs from malicious image inputs while maintaining the model’s original performance.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors present MLLM-Protector as a plug-and-play module for MLLMs that addresses malicious-image safety without compromising performance.

    Source: §9 · p. 9

    Detoxifying harmful responses adds computational overhead, motivating the design of more lightweight detoxifiers.

    Source: §10 · p. 9

Abstract

from arXiv · show

The deployment of multimodal large language models (MLLMs) has brought forth a unique vulnerability: susceptibility to malicious attacks through visual inputs. This paper investigates the novel challenge of defending MLLMs against such attacks. Compared to large language models (LLMs), MLLMs include an additional image modality. We discover that images act as a ``foreign language" that is not considered during safety alignment, making MLLMs more prone to producing harmful responses. Unfortunately, unlike the discrete tokens considered in text-based LLMs, the continuous nature of image signals presents significant alignment challenges, which poses difficulty to thoroughly cover all possible scenarios. This vulnerability is exacerbated by the fact that most state-of-the-art MLLMs are fine-tuned on limited image-text pairs that are much fewer than the extensive text-based pretraining corpus, which makes the MLLMs more prone to catastrophic forgetting of their original abilities during safety fine-tuning. To tackle these challenges, we introduce MLLM-Protector, a plug-and-play strategy that solves two subtasks: 1) identifying harmful responses via a lightweight harm detector, and 2) transforming harmful responses into harmless ones via a detoxifier. This approach effectively mitigates the risks posed by malicious visual inputs without compromising the original performance of MLLMs. Our results demonstrate that MLLM-Protector offers a robust solution to a previously unaddressed aspect of MLLM security.

1 Introduction

The introduction frames malicious image inputs as an under-addressed MLLM safety vulnerability: images can bypass safety awareness, while their continuous variation complicates alignment. It presents MLLM-Protector, a detector-and-detoxifier plug-in trained with Safe-Harm-10K, to mitigate harmful outputs while preserving model performance.

1 Introduction

  • Images can act as a “foreign language” whose malicious-query semantics bypass an MLLM’s safety awareness and trigger harmful content.The introduction reports a higher likelihood of harmful responses with image inputs than text inputs.Source: §1 · pp. 1–2
  • Aligning image inputs is difficult because continuous image variation expands the input space, while exhaustive safety tuning risks catastrophic forgetting of original capabilities.The paper contrasts continuous images with discrete text tokens and notes that MLLMs use fewer image-text training data than text pretraining data.Source: §1 · p. 2
  • MLLM-Protector divides the task into a lightweight harm detector for MLLM responses and a detoxifier that rewrites responses identified as potentially harmful.The components can be trained independently and integrated as a plug-and-play module.Source: §1 · p. 2
  • MLLM-Protector mitigates harmful outputs from malicious image inputs while maintaining the model’s original performance.The paper presents it as a plug-and-play defense paradigm for MLLMs.Source: §1 · pp. 2–3
  • Safe-Harm-10K is a synthesized dataset pairing each question across varied topics with safe and harmful responses for training the detector and detoxifier.The paper also identifies analysis of malicious-image vulnerability and the difficulty of addressing it with SFT as contributions.Source: §1 · p. 2

2 Related Work

Related work describes MLLMs’ visual-dialogue capabilities alongside their susceptibility to malicious visual inputs, and surveys attacks and defenses developed for LLMs. It also situates the paper among alignment approaches and identifies common malicious-input scenarios used in the paper.

2 Related Work

  • MLLMs support dialogue based on visual inputs, but current state-of-the-art models are more prone to malicious visual inputs.The related-work discussion distinguishes these visual-input risks from the capabilities of MLLMs.Source: §2 · p. 3
  • LLM attacks include malicious user jailbreaks, prompt leakage, prompt injection, and third-party indirect prompt injection; this work targets malicious image inputs.Existing LLM defenses include self-reminders, input detection, and in-context learning.Source: §2 · p. 3
  • Table 1 lists six common malicious-input scenarios and their abbreviations, while Table 2 describes perplexity comparisons by response type and input modality.The Table 2 caption reports lower harmless-response perplexity for text-only inputs and more frequent harmful content with image inputs.Source: Table 1 · p. 3, Table 2 · p. 3
  • Alignment approaches reviewed include RLHF and SFT, visual-signal interpretation, response bootstrapping with RRHF or RAFT, and preference-weighted DPO.The passage describes PPO as a method used by RLHF to maximize output reward.Source: §2 · p. 3

3 Observation

Malicious visual inputs make instruction-tuned MLLMs more prone to harmful responses than text-only inputs, while preliminary SFT yields marginal safety gains and can increase attack success rates. The section examines the limits of SFT and its image-text training setup.

3 Observation

  • MLLMs can produce harmful responses to malicious images even when they can recognize and refuse similar text-only requests.The cited observation includes both related natural images and OCR images containing relevant phrases.Source: §3 · pp. 3–4
  • For text-only inputs, harmful responses have higher perplexity than harmless ones, but this pattern does not hold for image inputs.The authors hypothesize that images act like a foreign language because image inputs have received less safety alignment than text.Source: §3 · p. 4

4 Vanilla Safety Fine-tuning

  • The preliminary SFT dataset pairs SafeRLHF malicious queries and annotated responses with generated query-related and OCR images, retaining harmless responses as targets.The resulting collection contains approximately 60,000 image-text pairs.Source: §4 · p. 4
  • SFT produces only marginal safety gains on MMSafetyBench, and attack success rates rise after SFT in many scenarios.The authors attribute the difficulty of image alignment to the continuous nature of image inputs.Source: Table 3 · p. 4, §4 · p. 4
  • Image continuity makes comprehensive alignment across possible scenarios substantially harder than alignment over discrete text inputs.The authors note that covering the range of potential image scenarios is considerably more complex.Source: §4 · p. 4
  • Most MLLMs receive image-modality training only during fine-tuning, with less data and shorter training than text pretraining, leaving them susceptible to catastrophic forgetting on standard tasks.The authors identify catastrophic forgetting as a risk of conventional SFT or RLHF alignment.Source: §4 · pp. 4–5
  • Because safety standards can vary by scenario, the authors argue that a plug-and-play approach can accommodate cases where content allowed in one setting is prohibited in another.Their example contrasts sexual-health advice for medical purposes with content prohibited for children.Source: §4 · p. 5
  • The inference algorithm generates an MLLM response, checks it with a harm detector, detoxifies it when harmful, and carries the input and output into the next round.Detoxification is conditional on the detector identifying the response as harmful.Source: §4 · p. 5

5 MLLM-Protector

MLLM-Protector separates safety alignment into detecting harmful model responses and detoxifying those responses. Its independently trained components are applied at inference as a plug-and-play defense for MLLMs.

5 MLLM-Protector

  • MLLM-Protector uses a divide-and-conquer defense that detects harmful responses and corrects them without training the MLLM itself.The components are independently trained and used during inference.Source: §5 · p. 5

5.1 Components

  • The harm detector uses a pretrained LLM backbone with a one-dimensional output layer to identify harmful content.The paper says a small LLM can provide satisfactory identification performance.Source: §5.1 · p. 5
  • The response detoxifier takes the query and harmful response as input and generates a corrected, harmless response related to the query.The paper contrasts this with replacing every harmful response with a fixed refusal sentence.Source: §5.1 · p. 5

5.2 Training and Inference

  • Safe-Harm-10K pairs questions across diverse scenarios with safe and harmful responses, and training also uses annotated SafeRLHF data and harmless image-conversation examples.ChatGPT synthesizes Safe-Harm-10K; the additional sources include accepted/rejected answers and LLaVA instruction-tuning data.Source: §5.2 · p. 5
  • The detector is trained with binary cross-entropy on response inputs, while the detoxifier uses autoregressive language modeling to generate harmless corrections.The detoxifier is trained to produce a harmless answer from the user query and harmful response.Source: §5.2 · p. 6
  • At inference, the harm detector checks the MLLM output first, and the detoxifier modifies it only if it is identified as harmful.Figure 3 shows examples in which the protector removes harmful content and gives reasons for not answering.Source: §5.2 · p. 6, Figure 3 · p. 7

6 Experiments

Experiments evaluate MLLM-Protector on MM-SafetyBench and FigStep, including image-based attacks. The reported results show reduced attack success rates on both benchmarks.

6.1 Implementation Details

  • The harm detector uses Open-LLaMA-3B and the detoxifier uses LLaMA-7B; their reported training times are about one and three hours, respectively.Both components were trained for three epochs using LoRA on eight A40 GPUs.Source: §6.1 · p. 7

6.2 Experiments on MM-SafetyBench

  • On MM-SafetyBench, MLLM-Protector significantly decreases attack success rates across input types, with near-complete prevention for illegal activity and hate speech scenarios.The evaluated inputs include text-only prompts, Stable Diffusion images, OCR images, and combined SD+OCR images.Source: §6.2 · p. 7

6.3 Experiments on FigStep

  • On FigStep, MLLM-Protector reduces attack success rates to nearly zero in most scenarios, despite rates reaching nearly 100% for some unprotected LLaVA settings.The benchmark represents harmful instructions as images and uses benign text prompts.Source: §6.3 · p. 7, Table 5 · p. 7

7 Helpfulness with MLLM-Protector

The paper compares original LLaVA, safety-fine-tuned LLaVA, and LLaVA with MLLM-Protector on GQA and MMVet. It reports that safety fine-tuning reduces original capabilities, whereas the plug-and-play protector bypasses that potential degradation.

7 Helpfulness with MLLM-Protector

  • Safety fine-tuning significantly reduces LLaVA’s original capabilities on GQA and MMVet, while MLLM-Protector bypasses potential degradation because it does not train the MLLM.The paper attributes greater susceptibility to catastrophic forgetting in image capabilities to the comparatively limited image-modality fine-tuning data and iterations.Source: §7 · p. 8, Table 6 · p. 8

8 Ablation Study

The harm detector distinguishes harmful from harmless responses, and its accuracy improves with stronger pretrained LLMs, with gains diminishing as model size grows. The evaluation uses SafeRLHF and Image QA, with Table 7 comparing detector sizes.

8 Ablation Study

  • The evaluation combines Safe-Harm-10K with image-text instruction-tuning data as Image QA, then uses 7:3 training and validation splits for SafeRLHF and Image QA.The combined training splits train the detector, which is evaluated separately on the two datasets.Source: §8 · p. 8
  • The harm detector distinguishes harmful responses from harmless ones in its output scores.Figure 4 shows red bars for harmful responses and green bars for harmless responses.Source: §8 · p. 8, Figure 4 · p. 8
  • Stronger pretrained LLMs yield more accurate harm detectors, but the extra accuracy gain diminishes as model size grows.The authors attribute this to identification being easier than direct generation; the detector runs once per response, adding marginal inference cost.Source: §8 · p. 8, Table 7 · p. 8

9 Conclusion

MLLM-Protector uses a divide-and-conquer defense for MLLMs, combining harm detection and response detoxification. The paper presents it as a plug-and-play module that improves safety without compromising model performance.

9 Conclusion

  • MLLM-Protector applies a divide-and-conquer approach to defending MLLMs.It integrates a harm detector to identify potentially harmful outputs and a detoxifier to amend them.Source: §9 · p. 9
  • The method is presented as a plug-and-play module that combines harm detection with detoxification.The detector identifies potentially harmful outputs, and the detoxifier amends them.Source: §9 · p. 9
  • The paper concludes that MLLM-Protector improves MLLM safety without compromising performance.Source: §9 · p. 9

10 Limitations

MLLM-Protector lowers MLLMs’ safety risks but adds inference cost, particularly when the detoxifier rewrites harmful responses. The ethical-impact discussion frames the work as defense against malicious image queries to MLLMs.

10 Limitations

  • The detoxifier adds computational overhead when it rewrites harmful responses, motivating the search for lighter detoxifiers with strong capabilities.The harm detector’s one forward pass adds negligible cost by comparison.Source: §10 · p. 9

11 Ethical Impact

  • The paper identifies malicious image queries to MLLMs as the target of its defense work.The ethical-impact discussion places this aim within the broader effort to ensure AI safety.Source: §11 · p. 9

A SFT Data Curation

The SFT collection contains about 60,000 image-text pairs built from malicious queries and harmless answers, using generated images and OCR images. Harm-detector training combines SafeRLHF, ChatGPT-generated examples, and harmless LLaVA conversations into about 370,000 samples.

A SFT Data Curation

  • The SFT collection contains approximately 60,000 image-text pairs, retaining harmless responses as ground truths for malicious queries.For each query, the authors collect Stable-Diffusion-generated images and OCR images containing query keywords.Source: A SFT Data Curation · p. 12

B Training Data for Harm Detector

  • SafeRLHF supplies around 300,000 malicious-query samples with answers labeled as harmful or harmless for detector training.Source: B Training Data for Harm Detector · p. 12
  • ChatGPT-generated data covers six harmful-content areas, totaling around 10,000 triplets or 20,000 QA pairs.The areas include hate speech, physical harm, malware generation, pornography, fraud, and privacy.Source: B Training Data for Harm Detector · p. 12
  • The authors add 50,000 harmless LLaVA instruction-tuning conversations; detector training totals around 370,000 samples and differentiates harmful from harmless responses.Source: B Training Data for Harm Detector · p. 12

B.1 Training Data for Detoxifier

The detoxifier is fine-tuned on triplets pairing a harmful response and text query with a harmless response. Its data-generation prompts cover six harm categories.

B.1 Training Data for Detoxifier

  • Triplets <a_acc, a_rej, q> pair a harmful response and text query with a harmless response for detoxifier fine-tuning.The LLM is trained to produce the harmless response from the harmful response and query.Source: §B.1 · p. 12
  • The detoxifier uses only data items (1) and (2) from the harm-detector training data.Source: §B.1 · p. 12
  • Data-generation prompts cover pornography, fraud, physical harm, privacy, hate speech, and malware generation.Source: Table 8 · p. 13, Table 9 · p. 13, Table 10 · p. 14
Loading 2401.02906v3…