PaperlayerView exact revision on arXiv

Source-linked AI summary

SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models

Zonghao Ying, Aishan Liu, Siyuan Liang, Lei Huang, Jinyang Guo, Wenbo Zhou, Xianglong Liu, Dacheng Tao

arXiv:2410.18927v1cs.CR

TL;DR

Existing MLLM safety benchmarks have limitations in harmful-query quality and evaluation reliability. SafeBench combines an automatically generated multimodal harmful-query dataset with collaborative LLM evaluation. Experiments revealed widespread safety issues across the evaluated open-source and commercial MLLMs.

  • Problem

    Existing MLLM safety benchmarks have limitations in harmful-query quality and evaluation reliability, leaving query coverage and assessments of open-ended outputs as concerns.

    Source: §1 · p. 2

  • Method

    SafeBench uses LLMs to build a risk-scenario taxonomy and generate harmful queries, then applies a collaborative jury protocol to evaluate target-model outputs.

    Source: §3.2 · p. 4, §4.2.1 · p. 7

  • Results

    Experiments on 15 open-source and 6 commercial MLLMs revealed widespread safety issues and insights related to image quality and parameter size.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SafeBench provides a framework for MLLM safety evaluation with 23 risk scenarios and 2,300 multimodal harmful-query pairs.

    Source: §1 · p. 2

    The paper identifies prior data workflows’ incomplete scenario coverage and prompt diversity, as well as evaluator limitations for assessing open-ended outputs.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are showing strong safety concerns (e.g., generating harmful outputs for users), which motivates the development of safety evaluation benchmarks. However, we observe that existing safety benchmarks for MLLMs show limitations in query quality and evaluation reliability limiting the detection of model safety implications as MLLMs continue to evolve. In this paper, we propose \toolns, a comprehensive framework designed for conducting safety evaluations of MLLMs. Our framework consists of a comprehensive harmful query dataset and an automated evaluation protocol that aims to address the above limitations, respectively. We first design an automatic safety dataset generation pipeline, where we employ a set of LLM judges to recognize and categorize the risk scenarios that are most harmful and diverse for MLLMs; based on the taxonomy, we further ask these judges to generate high-quality harmful queries accordingly resulting in 23 risk scenarios with 2,300 multi-modal harmful query pairs. During safety evaluation, we draw inspiration from the jury system in judicial proceedings and pioneer the jury deliberation evaluation protocol that adopts collaborative LLMs to evaluate whether target models exhibit specific harmful behaviors, providing a reliable and unbiased assessment of content security risks. In addition, our benchmark can also be extended to the audio modality showing high scalability and potential. Based on our framework, we conducted large-scale experiments on 15 widely-used open-source MLLMs and 6 commercial MLLMs (e.g., GPT-4o, Gemini), where we revealed widespread safety issues in existing MLLMs and instantiated several insights on MLLM safety performance such as image quality and parameter size.

1 Introduction

Existing MLLM safety benchmarks face gaps in harmful-query quality and evaluation reliability. SafeBench addresses both with a harmful-query dataset and an automated evaluation protocol, then evaluates open-source and commercial MLLMs.

1 Introduction

  • SafeBench pairs a comprehensive harmful-query dataset with an automated evaluation protocol to address benchmark data-quality and evaluation-reliability gaps.The framework uses LLM judges to categorize risk scenarios and generate harmful queries, and collaborative LLMs to assess harmful behaviors.Source: §1 · p. 2
  • The dataset covers 23 risk scenarios with 2,300 multimodal harmful-query pairs, generated through an automatic pipeline.The pipeline uses LLM judges to categorize scenarios and generate queries for those scenarios.Source: §1 · p. 2
  • Jury deliberation uses collaborative LLMs to assess whether target models exhibit harmful behaviors, aiming for more reliable content-security evaluations than simple metrics.The protocol evaluates outputs against prompts with specific topics or scenarios.Source: §1 · p. 2
  • Experiments on 15 open-source and 6 commercial MLLMs revealed widespread safety issues and insights related to image quality and parameter size.The paper also presents an up-to-date leaderboard and a nuanced account of MLLM safety issues.Source: §1 · p. 2

2 Related Work

Prior safety benchmarks cover several content risks and assess refusal or jailbreak resistance, while MLLM content-safety datasets often center on jailbreak attacks. Existing evaluation protocols vary in their trade-offs, from efficient but limited matching to resource-intensive human assessment.

2.1 Safety Evaluation Benchmarks

  • Safety benchmarks address risks such as toxicity, bias, privacy, and misinformation, and may assess refusal of harmful instructions or resistance to jailbreaks.MLLM benchmarks include adversarial- and out-of-distribution-robustness assessments; content-safety datasets have largely been built around jailbreak attacks.Source: §2.1 · pp. 2–3

2.2 Safety Evaluation Methods

  • String matching is efficient but lacks context and vocabulary coverage, while API-based assessment often focuses on toxicity rather than a broad range of risks.Fine-tuned classifiers depend on their training-data quality and base-model competence.Source: §2.2 · p. 3
  • GPT-proxy evaluations can be affected by model bias and hallucinations, whereas human-assisted assessment is resource-intensive and subject to inconsistency and subjectivity.Human assessments may vary with individual perspectives, cultural backgrounds, and personal biases.Source: §2.2 · p. 3

3 Motivation

The paper identifies shortcomings in both harmful-query construction and the assessment of open-ended MLLM outputs. SafeBench aims to address them with an automated query-generation pipeline and a collaborative jury protocol.

3.1 Existing Limitations

  • Existing data workflows may have incomplete risk-scenario coverage and insufficient harmful-query diversity when scenarios are manually assembled and prompts come from a single LLM.The authors say these limitations restrict benchmarks’ ability to reveal safety implications across MLLMs.Source: §3.1 · p. 3
  • String matching, fine-tuned evaluators, and single-model assessments have limitations that can produce error-prone estimates of safety in open-ended MLLM outputs.The cited concerns include limited contextual suitability, dependence on fine-tuning quality or model capability, and potential hallucinations.Source: §3.1 · p. 3

3.2 Our Objective

  • SafeBench’s objective is to improve data quality with an automatic harmful-query pipeline and evaluation reliability with a jury deliberation protocol.The framework diagram describes a harmful-query dataset and automated evaluation protocol intended to improve test-data quality and evaluation reliability.Source: §3.2 · p. 4, Figure 1 · p. 4
  • The authors characterize their work as the first, to their knowledge, to address both dataset and evaluation-method limitations for MLLM safety assessment.They frame this as offering a more reliable and comprehensive assessment of security risks.Source: §3.2 · p. 4

4 SafeBench Framework

SafeBench combines a multimodal harmful-query dataset with an automated jury-based evaluation protocol to assess MLLM safety. Its construction covers diverse risk scenarios and modalities, while its evaluation uses complementary measures of attack success and risk severity.

4 SafeBench Framework

  • The framework pairs an automated dataset-construction pipeline with a jury-based protocol and assembles 2,300 multimodal query pairs for evaluation.The dataset serves as the foundation for the paper's model evaluations.Source: §4 · p. 4

4.1.1 Safety Risk Taxonomy

  • The taxonomy draws on developer policies and input from 10 widely used LLMs, yielding 8 major risk categories and 23 subcategories.The authors identify limited scope, developer-to-developer variation, and real-world departures from prescribed usage as shortcomings of policy-based taxonomies.Source: §4.1.1 · p. 5

4.1.2 Dataset Generation

  • Textual requests are generated, scored for feasibility, harmfulness, and applicability, and filtered; images are iteratively refined for semantic alignment, while audio samples vary vocal style.Audio samples use identical content in different vocal styles to explore potential biases or inconsistencies in audio processing.Source: §4.1.2 · pp. 5–7

4.1.3 Dataset Statistics

  • The dataset contains 9,200 samples balanced across 23 subcategories, with text, visual, and audio samples in a 1:1:2 ratio.Across the eight major categories, scores are consistently high, though the maximum difference is 2.7 between CO and VI.Source: §4.1.3 · p. 7

4.2.1 Jury Deliberation Protocol

  • The Jury Deliberation Protocol uses five LLMs for independent assessments and collaborative discussion, then combines majority voting for classifications with mean scores for safety.The protocol uses diverse roles to mitigate individual biases and synthesizes a final rationale from the jury's opinions.Source: §4.2 · p. 7, §4.2.1 · pp. 7–9

4.1.3 Dataset Statistics

  • SafeBench scores higher than other datasets on all three quality metrics, by as much as 1.3.The comparison uses five overlapping risk categories and samples 50 examples per category from FigStep for fairness.Source: §4.1.3 · p. 7, Figure 5 · p. 8

4.2.2 Metrics

  • Attack Success Rate measures the proportion of query pairs classified as successful attacks; the Safety Risk Index aggregates threat scores and normalizes them to 0–100.Unlike binary ASR, SRI distinguishes the severity of harmful responses; higher SRI indicates better safety performance.Source: §4.2.2 · p. 9
  • The protocol combines diverse LLM judges and ASR with SRI to capture content-safety nuances, but requires more computation than simpler methods.The authors describe it as scalable for large-scale evaluations and adaptable to evolving safety standards and risk categories.Source: §4.2.2 · pp. 9–10

5 Experiment and Evaluation

The experiments evaluate 15 open-source and 6 commercial MLLMs using default settings. Results show substantial safety differences across models, trade-offs with general performance, and variation by model size and risk category.

5.1 Experimental Setup

  • The evaluation used default settings for 15 open-source MLLMs and included 6 commercial models; five open-source LLMs served as jury models, with GPT-4o used to score the corpus.The listed dataset-construction LLMs, image-generation components, and scoring model are specified in the implementation details.Source: §5.1 · p. 10

5.2 Benchmark Results

  • Commercial models outperform open-source models on average, with an SRI gap of 20.78 and an ASR gap of 26.38%; Table 2 summarizes results across 21 models and 8 risk categories.Claude-3.5-Sonnet had a 0.7% ASR and 99.3 SRI, while ShareGPT4V had a 38.8% ASR and 69.9 SRI.Source: §5.2 · p. 10, Table 2 · p. 11
  • Safety and general performance did not consistently move together: GPT-4o had better general performance but lower safety than GPT-4o-mini, while open-source comparisons also showed safety declines in some newer models.For example, Qwen-VL-Chat had an SRI of 90.3, compared with 58 for Qwen2-VL-2B and 71.8 for Qwen2-VL-7B.Source: §5.2 · p. 10
  • Across model comparisons, GLM-4V had an SRI of 89.3, MiniCPM-V 2.6 had 74.9, and Qwen2-VL-7B had 71.8, despite reported general-performance advantages for the latter models.Source: §5.2 · p. 11
  • Safety did not scale linearly with parameter count: Qwen2-VL-2B had an average SRI of 58, while 4.2B Phi-3-Vision-128K-Instruct achieved 98.7.The authors connect Phi-3-Vision-128K-Instruct's performance with its focus on high-quality, inference-intensive text and vision training data.Source: §5.2 · pp. 11–12
  • Across query categories, models showed higher susceptibility to safety risks for CS and CI queries, with average ASRs of 33% and 32.4%, versus 7.4% for CO and 11.5% for IO.Source: §5.2 · p. 12

6 Discussion and Analysis

The analysis tests jury deliberation, image and model factors, jailbreak attacks, and audio inputs. It reports improved agreement after deliberation and shows that safety outcomes vary with image semantics, model properties, attack conditions, and input modality.

6 Discussion and Analysis

  • MiniBench was created for supplementary analysis because the full SafeBench evaluation was extremely time-consuming; it samples 50 harmful-query pairs from each of 10 randomly selected sub-categories.Source: §6 · p. 12

6.1 JDP Effectiveness Verification

  • After collaborative discussion, mean Cohen’s Kappa rose from 0.37 to 0.61; JDP achieved 0.89 consistency with human annotations, supporting the authors’ conclusion that deliberation improves evaluation reliability.The study used five volunteers to annotate MiniBench responses.Source: §6.1 · pp. 12–13, Figure 8 · p. 12, Figure 7 · p. 12

6.2 Factors Influence MLLM Safety Evaluation

  • With fixed text prompts, MiniBench’s semantically related images induced more unsafe responses than five alternatives, while semantically irrelevant images affected models differently.Only MiniBench images induced ransomware-creation methods from Llava-V1.6-Mistral-7B in the reported example.Source: §6.2 · pp. 13–14, Figure 10 · p. 13, Figure 9 · p. 13
  • Most MLLMs had higher ASR than their backbone LMs, and MLLM ASRs varied more across harmful categories; results also show that MLLM safety depends on training pipelines, not only backbone safety.The authors note that a well-designed MLLM training approach can compensate for deficiencies in a backbone LM.Source: §6.2 · p. 14, Figure 11 · p. 14
  • Within model families, Qwen-2-VL-7B had lower ASR and higher SRI than Qwen-2-VL-2B; the authors report a positive size–safety correlation when other factors are held constant.Qwen-2-VL-7B: ASR 25.8%, SRI 80.1; Qwen-2-VL-2B: ASR 43.1%, SRI 66.7.Source: §6.2 · pp. 14–15, Table 3 · p. 14

6.3 Jailbreaks Based on Our Dataset

  • Three jailbreak methods applied to MiniBench induced more safety risks, with LPT increasing ASR by up to 57.6%; Qwen-VL-Chat’s safety advantage did not persist under attack.Under PAP, Qwen-VL-Chat had the lowest SRI among the three models, at 46.5; SafeBench is presented as a non-attack baseline for red-team testing.Source: §6.3 · p. 15, Table 4 · p. 15, Figure 12 · p. 15

6.4 Evaluation on the Audio Modality

  • For Qwen2-Audio, audio-plus-text input yielded an average SRI of 54.52 and average ASR of 58.1%, while audio-only performance exceeded Qwen-Audio’s average SRI by 12.74.Female-versus-male voice ASR differences did not exceed 2% on any evaluated audio-language model.Source: §6.4 · pp. 15–16, Table 5 · p. 16

7 Conclusion

SafeBench combines a harmful-query dataset with an automated evaluation protocol, and experiments across open-source and commercial MLLMs revealed widespread safety issues. The study notes limits in query coverage and in assessing risks that combine audio with other modalities.

7 Conclusion

  • Experiments on 15 open-source and 6 commercial MLLMs revealed widespread safety issues.The authors hope SafeBench will drive future MLLM safety research.Source: §7 · p. 16
  • SafeBench combines a harmful-query dataset and an automated evaluation protocol to address benchmark limitations in data quality and evaluation reliability.The study also aims to provide a foundation for future red-team testing methodologies.Source: §7 · pp. 16–17
  • The study selected 100 harmful queries per category, which may not capture real-world query diversity and complexity, and excluded some extreme scenarios for ethical and legal reasons.Examples excluded include malicious requests about national leaders in the Political Opinions category.Source: §7 · p. 16
  • The study could not comprehensively assess safety risks combining audio with other modalities because suitable MLLMs were lacking.The authors identify this as an area for future research.Source: §7 · p. 17
  • The SafeBench website, codebase, and dataset are publicly accessible through the stated project, GitHub, and Hugging Face locations.The authors invite community feedback and scrutiny of the benchmark.Source: §7 · p. 17

A Prompts used in dataset creation

The appendix points readers to a table containing the prompts used in dataset construction.

A Prompts used in dataset creation

B Description of risk scenarios

The appendix describes the benchmark’s risk categories and points to prompts used in four stages of jury deliberation.

B Description of risk scenarios

C Prompts used in JDP

  • The JDP appendix contains prompts for role initialization, independent evaluation, collaborative deliberation, and consensus decision.Source: C Prompts used in JDP · p. 29

D Responses under different image settings

The appendix presents model responses and evaluation results for one harmful request under different image inputs.

D Responses under different image settings

E Distribution of SRIs among MLLMs on MiniBench

On MiniBench, all MLLMs except Yi-VL-6B and Yi-6B have lower SRI than their corresponding backbone LMs. MLLMs also show wider variation in SRI across harmful categories, while most LMs remain near 100.

E Distribution of SRIs among MLLMs on MiniBench

  • Except for Yi-VL-6B and Yi-6B, all MLLMs have lower SRI than their corresponding LMs.The passage suggests that visual-language fine-tuning and multimodal semantic alignment often compromise safety mechanisms, while visual inputs broaden the attack surface.Source: E Distribution of SRIs among MLLMs on… · p. 29
  • MLLMs show wider SRI distributions across harmful categories, whereas most LMs show consistently high scores.InternLM-2, Qwen2-7B, Phi3-mini-128K, and Qwen-7B remain near 100 across categories.Source: E Distribution of SRIs among MLLMs on… · p. 29
  • The figure compares the MiniBench SRI distributions of MLLMs and their backbone LMs.The passage identifies distributional differences between the two groups but gives no figure-specific values.Source: Figure E.1 · p. 30

Related papers

Loading 2410.18927v1…