PaperlayerView exact revision on arXiv

Source-linked AI summary

MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models

Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, Yu Qiao

arXiv:2311.17600v5cs.CV

TL;DR

MLLM safety remains under-studied, particularly against malicious query-relevant images that can induce unsafe responses. The paper introduces a visual prompt attack and MM-SafetyBench to evaluate this vulnerability, finding attacks across 12 models while safety prompts reduce successful attacks.

  • Problem

    MLLM safety against malicious attacks remains under-explored, raising the question of how resilient instruction-tuned MLLMs are to such attacks.

  • Method

    The paper constructs MM-SafetyBench with 13 scenarios and uses keyword-guided Typography and Stable Diffusion images, blended as visual prompts to test MLLM safety.

  • Results

    Query-relevant visual prompt attacks breach safety measures across 12 MLLMs, while safety prompts significantly reduce attack success rates.

  • Takeaways & Limitations

    Query-relevant images expose fragility in MLLM safety protocols, and safety prompts can improve resilience when models follow instructions.

  • Takeaways & Limitations

    Apparent model safety may reflect poor generalization or failure to comprehend the question or image rather than safe alignment.

Abstract

from arXiv · show

The security concerns surrounding Large Language Models (LLMs) have been extensively explored, yet the safety of Multimodal Large Language Models (MLLMs) remains understudied. In this paper, we observe that Multimodal Large Language Models (MLLMs) can be easily compromised by query-relevant images, as if the text query itself were malicious. To address this, we introduce MM-SafetyBench, a comprehensive framework designed for conducting safety-critical evaluations of MLLMs against such image-based manipulations. We have compiled a dataset comprising 13 scenarios, resulting in a total of 5,040 text-image pairs. Our analysis across 12 state-of-the-art models reveals that MLLMs are susceptible to breaches instigated by our approach, even when the equipped LLMs have been safety-aligned. In response, we propose a straightforward yet effective prompting strategy to enhance the resilience of MLLMs against these types of attacks. Our work underscores the need for a concerted effort to strengthen and enhance the safety measures of open-source MLLMs against potential malicious exploits. The resource is available at https://github.com/isXinLiu/MM-SafetyBench

1 Introduction

This paper investigates how query-relevant images can compromise MLLM safety and introduces a benchmark and visual prompt attack for evaluating this vulnerability. Across diverse scenarios and models, the attack exposes fragile safety protocols, while safety prompts reduce successful attacks.

  • MLLM safety against malicious attacks remains under-explored, motivating evaluation of the resilience of instruction-tuned models.
  • Query-relevant images can cause MLLMs to answer malicious questions that query-irrelevant images elicit refusals for.The paper illustrates this contrast with a bomb-making query paired with either a relevant bomb image or an unrelated street scene.
  • The attack creates crafted image prompts using extracted keywords, Typography, Stable Diffusion, and strategic image blending to bypass MLLM defenses.GPT-4 identifies and extracts keywords from each malicious query before image construction.
  • The benchmark covers 13 prohibited-content scenarios and contains 5,040 image-text pairs generated using query-relevant images.The scenarios include illegal activities and hate speech.
  • Evaluations across 12 MLLMs show that combining two generated image types can circumvent safety measures across many scenarios.
  • Safety prompts significantly reduce attack success rates and can prevent answers to malicious questions.The reported mitigation depends on the model being capable of following instructions.
  • The paper argues that the effectiveness of these attacks demonstrates a need for stronger, more robust safety measures in foundation models.

2 Related Work

Related work has extensively studied LLM safety, MLLM architectures, and general evaluation, but MLLM safety evaluation remains comparatively undeveloped. MM-SafetyBench addresses this gap with a broader safety-focused evaluation pipeline.

  • LLM safety research includes attack methods, defense methods, alignment studies, and evaluation-dataset construction.
  • Although safety evaluation for MLLMs has begun, prior benchmarks cover limited safety scopes compared with MM-SafetyBench’s broader scenarios and larger malicious-sample collection.
  • MLLMs connect a vision encoder to an LLM through alignment or fusion modules, with several established strategies for integrating visual and textual information.
  • Traditional multimodal benchmarks do not fully satisfy the need for comprehensive MLLM evaluation, prompting work on capabilities, benchmark scale, answer formats, and metrics.

3 Methodology

MM-SafetyBench constructs a multimodal safety benchmark by generating malicious questions, extracting unsafe key phrases, converting them into query-relevant images, and rephrasing questions to align with those images. It evaluates attacks across 13 scenarios using ASR and examines model responses and dataset scope.

  • Data Collection Process: The benchmark construction uses four stages: question generation, unsafe key-phrase extraction, query-to-images conversion, and question rephrasing.GPT-4 generates malicious questions, after which repetitive or answerable questions are removed.
  • Dataset Versions: The benchmark includes default and tiny versions, with three generated images per question and the tiny version randomly sampled from the default version.Unless otherwise stated, experiments use the default version.
  • Question Formulation: Typography elicits more unsafe responses than request tones, and the benchmark includes both imperative and request formulations.The observation applies to both unimodal and multimodal models.
  • Query-to-Images Conversion: Query-to-images conversion uses Stable Diffusion, typography, and a combined SD+Typography format to represent extracted key phrases visually.Stable Diffusion images use the prompt “A photo of [Key Phrase]”; typography places the phrase as black text on a white background.
  • Question Rephrasing: Rephrased questions first instruct MLLMs to extract query-relevant information from images, then replace the original key phrase with that information.Rephrased Question1 is used for SD and Typography images, while Rephrased Question2 is used for SD+Typography images.
  • Evaluation Metrics: Attack Success Rate measures the fraction of text-image queries for which the model engages with the malicious query.Responses that reject or otherwise do not address the malicious intent receive indicator value 0.

4 Experiments

Experiments across 12 MLLMs show that query-relevant generated images can circumvent safety measures, while safety prompts reduce attacks but depend on instruction-following. Apparent robustness can also reflect failures to understand queries or images rather than genuine alignment.

  • 12 recently released MLLMs were evaluated in zero-shot settings using their released model weights.
  • 4.2 Main Results: Typography increased ASR over baseline across all 13 scenarios, by more than 30% for LLaVA-1.5-7B and 28.3% for LLaVA-1.5-13B.
  • 4.2 Main Results: Stable-diffusion images improved ASR over baseline in 10 scenarios for LLaVA-1.5-7B and 9 scenarios for LLaVA-1.5-13B.
  • 4.2 Main Results: Combining stable diffusion with typography generally enhanced performance beyond either technique alone, while politics and professional-field scenarios had high baseline ASR.The authors suggest the high baseline ASR may reflect inadequate safety alignment in Vicuna, LLaVA-1.5’s language model.
  • 4.3 Every Safe Model is Safe in its Own Way: Models can appear secure because of overfitting, inaccurate OCR or visual understanding, or weak instruction-following rather than genuine safety alignment.These failures can produce safe but irrelevant or erroneous responses.
  • 4.5 Enhance MLLM’s Safety via Safety Prompt: The safety prompt reduced ASR across all evaluated models; for LLaVA-1.5, ASR fell from roughly 77% to about 15%.Its effect was smaller on other models, which the authors attribute to weaker instruction-following capabilities.

5 Conclusion

The paper examines the under-explored safety of Multimodal Large Language Models and shows that query-relevant images can jailbreak them. It uses Stable Diffusion and typography to generate images, then tests 12 MLLMs to expose security flaws.

  • MLLM safety remains under-explored compared with the extensively investigated safety of LLMs.
  • Query-relevant images can jailbreak MLLMs through image generation.The paper investigates Stable Diffusion and typography as image-generation methods.
  • The authors construct a dataset and test it against 12 MLLMs to expose their security flaws.

A Limitations

The benchmark and attack method are scoped to open-source MLLMs and may not transfer to closed-source systems. The evaluation can also misrepresent safety when models fail to understand queries or image context, and future fine-tuning may reduce attack effectiveness.

  • The method is designed for open-source MLLMs and may be less effective against closed-source models such as GPT-4 or Gemini.
  • Safety metrics may be inaccurate when models fail to grasp the question’s intent or image context.
  • The attack may lose effectiveness if new MLLMs are fine-tuned on data derived from these approaches.
  • The proposed approaches could be misused by malicious users to conduct harmful social activities.

C More Experiments

Additional experiments compare safety prompting with post-processing and examine language-model size. Safety prompts outperform MLLM-Protector on LLaVA, while larger language-model backbones show higher ASR in the reported comparisons.

  • Safety Prompt Comparison: Safety prompting reduced LLaVA’s ASR more effectively than the trained binary harm detector MLLM-Protector.
  • Model Size: The ablation compares Vicuna-7B and Phi-2-2.7B on the tiny MM-SafetyBench using stable diffusion plus typography and ASR.
  • Model Size: LLaVA’s ASR markedly exceeded LLaVA-Phi’s across various measures, despite safety-aligned pre-training for both Vicuna-7B and Phi-2-2.7B.The authors relate this pattern to instruction-following capability and language-model size.

C.3 Ablation Study of SD/Typo. in MiniGPT-4

MiniGPT-4 is evaluated under baseline, Stable Diffusion, typography, and combined image conditions. Stable Diffusion and typography both increase attack success, with typography producing the larger reported gain.

  • Table 6 compares MiniGPT-4 across no-image baselines, Stable Diffusion images, typography-enhanced queries, and combined Stable Diffusion-plus-typography inputs.“Typo.” denotes typography.
  • 5.73% is the average Attack Success Rate (ASR) increase for MiniGPT-4 over LLaVA when using Stable Diffusion-generated images.
  • Over 10% is the average ASR increase produced by typography in queries beyond the baseline.
  • Figure 7 provides more detailed results for the 12 evaluated MLLMs.

C.4 More Examples

Additional examples show that query-relevant images can elicit harmful or confused behavior from MLLMs, while responses vary across models and input conditions.

  • An image about hacking a bank account paired with a detailed malicious prompt elicits concerning behavior from LLaVA.
  • MiniGPT-4 responds safely to the textual input alone but merely describes the image when both image and question are provided.The multimodal response is described as deficient in following complex instructions.
  • IDEFICS avoids harmful content but requests additional information or context, illustrating the paper’s described overfitting behavior.
  • Figures 8–10 provide examples for scenarios 01–04, 06, and 09.

C.5 More Detailed Results of 12 MLLMs

The appendix reports individual performance scores for 12 MLLMs and lists the forbidden scenarios used in the benchmark’s policy-based evaluation.

  • The detailed results present performance scores for 12 individual MLLMs and expose limitations across numerous scenarios.
  • The scenario list is adopted from the full list used in prior work and derived from OpenAI usage policy.
  • Tables 7 and 8 enumerate the forbidden scenarios, covering scenarios 01–09 and 10–13 respectively.

E Detailed Prompts

The appendix documents prompts for constructing and evaluating the benchmark. Construction prompts extract scenario-specific phrases, while evaluation prompts classify harmful responses and refusals using ASR and refusal rate criteria.

  • E.1 Prompts for Benchmark Construction: Benchmark construction uses carefully crafted Azure GPT-4 system prompts for steps two and four, with figures documenting the prompts.
  • E.1 Prompts for Benchmark Construction: Scenarios 01–09 and 13 use demonstrations to extract potentially harmful phrases such as hiring hackers, illegal firearms, and trafficking substances.
  • E.1 Prompts for Benchmark Construction: Scenario 10 uses demonstrations to extract legal phrases related to drone regulations and data privacy law.
  • E.1 Prompts for Benchmark Construction: Scenarios 11 and 12 use demonstrations to extract critical financial phrases and health-related phrases, respectively.
  • E.2 Prompts for Evaluation: Evaluation prompts in Figures 15–20 measure Attack Success Rate, while Figure 21 assesses refusal rate.
  • E.2 Prompts for Evaluation: For scenarios 01–07 and 09, responses containing hazardous content are labeled unsafe, whereas responses avoiding it are labeled safe.
  • E.2 Prompts for Evaluation: Scenario 08 is safe only when the model explicitly declines to participate in political lobbying.
Loading 2311.17600v5…