PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSSBench: Is Your Multimodal Language Model Oversensitive to Safe Queries?

Xirui Li, Hengguang Zhou, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Cho-Jui Hsieh

arXiv:2406.17806v1cs.CLcs.AIcs.CRcs.CVcs.LG

TL;DR

MLLMs can refuse benign queries when visual stimuli prompt cognitive-distortion-like perceptions of harm, a behavior that can compromise user experience. The paper introduces MOSSBench, a benchmark of 300 reviewed image-text queries, and evaluates it across 20 MLLMs. Oversensitivity is widespread, with refusal rates reaching 76.33% for Claude-3 Opus (web), and the authors report that safer models may be more oversensitive.

  • Problem

    How prevalent is MLLM oversensitivity to benign multimodal queries, and which response-process failures contribute to it?

    Source: §1 · p. 2

  • Method

    The authors construct MOSSBench, a 300-pair benchmark of benign image-text queries, and evaluate it across 20 MLLMs with failure analysis.

    Source: §1 · p. 2

  • Results

    Oversensitivity is widespread; Claude-3 Opus (web) rejects 76.33% of benign queries, and safer models may be more oversensitive.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings motivate refined safety mechanisms that balance caution with context-appropriate responses to improve MLLM reliability.

    Source: §7 · p. 9

    MOSSBench covers only a small fraction of potential oversensitivity-triggering situations, and the underlying mechanisms remain poorly understood.

    Source: A Limitations · p. 15

Abstract

from arXiv · show

Humans are prone to cognitive distortions -- biased thinking patterns that lead to exaggerated responses to specific stimuli, albeit in very different contexts. This paper demonstrates that advanced Multimodal Large Language Models (MLLMs) exhibit similar tendencies. While these models are designed to respond queries under safety mechanism, they sometimes reject harmless queries in the presence of certain visual stimuli, disregarding the benign nature of their contexts. As the initial step in investigating this behavior, we identify three types of stimuli that trigger the oversensitivity of existing MLLMs: Exaggerated Risk, Negated Harm, and Counterintuitive Interpretation. To systematically evaluate MLLMs' oversensitivity to these stimuli, we propose the Multimodal OverSenSitivity Benchmark (MOSSBench). This toolkit consists of 300 manually collected benign multimodal queries, cross-verified by third-party reviewers (AMT). Empirical studies using MOSSBench on 20 MLLMs reveal several insights: (1). Oversensitivity is prevalent among SOTA MLLMs, with refusal rates reaching up to 76% for harmless queries. (2). Safer models are more oversensitive: increasing safety may inadvertently raise caution and conservatism in the model's responses. (3). Different types of stimuli tend to cause errors at specific stages -- perception, intent reasoning, and safety judgement -- in the response process of MLLMs. These findings highlight the need for refined safety mechanisms that balance caution with contextually appropriate responses, improving the reliability of MLLMs in real-world applications. We make our project available at https://turningpoint-ai.github.io/MOSSBench/.

1 Introduction

The paper frames MLLM refusals of benign queries as oversensitivity resembling human cognitive distortions, and asks how prevalent the behavior is and which response-process failures contribute. It introduces MOSSBench and reports widespread oversensitivity across evaluated models.

1 Introduction

  • The paper identifies Exaggerated Risk, Negated Harm, and Counterintuitive Interpretation as visual stimuli that can trigger unwarranted refusals of benign queries.These refusals can prevent models from adequately responding to user queries and compromise user experience.Source: §1 · p. 2
  • The paper asks how prevalent oversensitivity is in current MLLMs and which response-process failures contribute to it.It describes MLLMs as exhibiting cognitive-distortion-like behavior when benign queries are perceived as harmful.Source: §1 · p. 2
  • MOSSBench contains 300 manually generated and filtered image-text VQA pairs across the three stimulus categories, reviewed by AMT evaluators for harmlessness and contextual appropriateness.The study evaluates 20 MLLMs and analyzes failures at perception, intent reasoning, and safety decision-making stages.Source: §1 · p. 2
  • Claude-3 Opus (web) and Gemini-pro 1.5 reject 76.33% and 63.67% of benign queries, respectively, while safer models may be more oversensitive.The authors also report that different visual stimuli tend to affect distinct stages of model reasoning.Source: §1 · p. 3

2 Related work

Prior safety-alignment benchmarks test whether models reject harmful prompts, while oversensitivity to benign requests has received less study. The paper positions MOSSBench as the first systematic MLLM oversensitivity benchmark.

2 Related work

  • Existing LLM and MLLM safety-alignment benchmarks evaluate models’ ability to reject harmful prompts.Safety alignment measures include RLHF, red teaming, and content filtering, but can produce undesired behaviors.Source: §2 · p. 3
  • The authors describe oversensitivity—rejecting benign requests—as under-explored, with the existing benchmark limited to language modality.Source: §2 · p. 3
  • The paper presents its study as the first systematic investigation of MLLM oversensitivity, broadening safety-alignment evaluation beyond harmful-prompt rejection.Source: §2 · p. 3

3 Visual stimuli of model oversensitivity

The section distinguishes three visual-stimulus patterns that can lead MLLMs to refuse benign queries: exaggerated risk cues, overlooked harm-negating context, and unlikely harmful interpretations.

3 Visual stimuli of model oversensitivity

  • The three categories describe visual stimuli that can trigger model refusals, including when the user’s query is benign.The examples distinguish apparent danger, explicit or contextual negation, and misinterpretation of intent.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Exaggerated Risk

  • Exaggerated Risk describes safety-alerting visual elements that pose little or no actual risk to the user’s request.Examples include alarming locations on a map or an inanimate dinosaur exhibit, whose apparent threat can overshadow the benign context.Source: §3.1 · pp. 3–4

3.2 Negated Harm

  • Negated Harm involves harmful objects or behavior shown in a context that opposes them, such as a prohibition sign over domestic violence.The model may focus on the violent scene and overlook the image’s positive, prohibitive context, then refuse related queries.Source: §3.2 · p. 4

3.3 Counterintuitive Interpretation

  • Counterintuitive Interpretation occurs when a model selects an unlikely harmful reading over the reasonable benign interpretation, such as placing a girl rather than her parrot in a cage.Source: §3.3 · p. 4

4 MOSSBench - Multimodal OverSenSitivity Benchmark

MOSSBench is introduced as a systematic benchmark for MLLM oversensitivity, built through a generation-and-filtering pipeline. It includes benign samples and a complementary harmful set, with dataset statistics organized around stimulus and scenario categories.

4 MOSSBench - Multimodal OverSenSitivity Benchmark

  • MOSSBench is presented as the first benchmark for systematically evaluating MLLM oversensitivity.Source: §4 · p. 4

4.3 Dataset statistics

  • MOSSBench contains 300 samples spanning diverse oversensitivity stimuli and everyday applications, with statistics covering stimulus types and scenario categories.The scenario labels follow HarmBench’s categorization protocol.Source: Table 1 · p. 5, §4.3 · p. 5, Figure 3 · p. 5

4.1 Samples generation

  • LLMs generate candidate scenarios and images, after which researchers manually remove unnatural or harmful cases and AMT reviewers assess harmlessness.Five annotators majority-voted 9% of samples as unanswerable without harmful content; the average harmfulness score was 1.84 on a 1-to-5 scale.Source: §4.1 · p. 5

4.2 Samples contrasting

  • The complementary harmful-sample set adds explicit malice to oversensitivity scenarios but is insufficient as a standalone safety benchmark.Source: §4.2 · p. 5

5 Assessing oversensitivity in MLLMs: an empirical study with MOSSBench

Across 20 MLLMs, MOSSBench reveals substantial refusal of benign queries, especially by some proprietary models. The study also finds higher oversensitivity among safer models, variation by stimulus and prompt, and differences between web and API versions.

5.1.1 Models of choice

  • The study covered 20 widely used models, including proprietary APIs and web versions and seven selected open-source models.The authors prioritized instruction-tuned variants among open-source models and note that their selection could not cover all available models.Source: §5.1.1 · p. 6

5.1.2 Protocols for human and automated evaluation

  • Human and automatic evaluations classify direct attempts to answer as compliance and explicit inability to answer as refusal, with 97.8% agreement among human raters.The automatic evaluation complements the human study to enable more scalable refusal-rate assessment.Source: §5.1.2 · p. 6

5.2 Highlighted results and findings

  • 76.33% and 63.67% of benign queries were refused by Claude 3 Opus (web) and Gemini Advanced, while GPT-series refusal rates were below 10%.Among open-source models, Qwen-VL-chat had the highest average refusal rate at 16%, followed by InternLM at 15.33%.Source: §5.2 · p. 6, Table 2 · p. 7
  • Claude 3 Opus refusal rose from 30.67% via API to 76.33% on the web, and Gemini rose from 34.67% to 63.67%.The authors suggest stricter policies or additional web-interface filters as possible contributors, but say the hypotheses are difficult to verify because the models are proprietary.Source: Figure 4 · p. 7
  • Higher safety levels were associated with greater refusal of benign queries, and proprietary models refused more Negated Harm and Counterintuitive Interpretation cases than Exaggerated Risk cases.Average refusal rates were 24.45% for Negated Harm, 24.60% for Counterintuitive Interpretation, and 17.35% for Exaggerated Risk; the authors describe the safety relationship as a potential trade-off.Source: §5.2 · p. 8
  • Safety-focused system prompts significantly increased Gemini-Pro 1.5’s refusal rate, while scrutiny- and helpfulness-focused prompts had limited impact.The comparison used a vanilla empty system prompt and prompts focused on helpfulness, scrutiny, or safety.Source: §5.2 · p. 8, Figure 5 · p. 8

6 Tracing the refusal behaviors to the reasoning process

The authors trace refusals by examining perception, intent reasoning, and safety judgement in Claude-3-opus and Gemini. Their analysis finds that error-prone stages differ across stimulus types, while stochasticity makes those stages difficult to isolate fully.

6.1 Dividing MLLM’s response process into three stages

  • The analysis separates perception, intent reasoning, and safety judgement, then uses oracle captions and explicit oracle intentions to probe errors at each stage.If a model rejects despite both oracle inputs, the authors treat the error as likely lying in safety judgement.Source: §6.1 · pp. 8–9
  • On 30 refusal-triggering examples, Claude-3-opus and Gemini had an average caption error rate of 35%, and some cases received no caption.The study randomly sampled 30 examples, with 10 from each stimulus type.Source: §6 · p. 8, §6.1 · p. 9, Figure 6 · p. 9
  • For Exaggerated Risk, supplying an accurate caption often mitigated oversensitivity, whereas Negated Harm and Counterintuitive Interpretation errors mostly persisted at the final decision stage.In the latter cases, models still rejected queries despite clarification of the image and user intent.Source: §6.1 · p. 9
  • Because MLLMs are stochastic, the authors say fully isolating the three response stages is inherently challenging.Figure 6 categorizes responses as correct perception, safety-critical misperception, or refusal across the three stimulus types; Figure 7 reports refusal rates for intent reasoning and safety judgement.Source: §6.1 · p. 9, Figure 6 · p. 9, Figure 7 · p. 9

7 Conclusion

MOSSBench evaluates MLLM oversensitivity using 300 everyday-scenario samples, and studies of 20 models find oversensitivity to be widespread.

7 Conclusion

  • MOSSBench uses 300 everyday-scenario samples to assess MLLM sensitivity levels.The authors describe the benchmark as an empirical study of oversensitivity across 20 MLLMs.Source: §7 · p. 9
  • Empirical studies on 20 MLLMs demonstrate that oversensitivity is a widespread concern.The conclusion frames this as an issue in MLLMs rather than a finding limited to a single model.Source: §7 · p. 9
  • The authors call for refined safety mechanisms that balance caution with context-appropriate responses to improve MLLM reliability.Source: §7 · p. 9

A Limitations

The authors describe MOSSBench as an initial investigation with limits in coverage and scalability, and argue that further evaluation and mechanism research are needed. They also discuss risks from the benchmark's scenarios and mitigation measures.

A Limitations

  • MOSSBench covers only a small fraction of potential oversensitivity-triggering situations, and human filtering and selection limit scalability.The authors describe it as an initial investigation using representative cases.Source: A Limitations · p. 15
  • The authors say their understanding of oversensitivity's underlying mechanisms and causes remains limited.They identify broader stimuli types and visual contexts as targets for future benchmark iterations.Source: A Limitations · p. 15

B Broader impact

  • The paper presents MOSSBench as a way to expose safety-alignment discrepancies and calls for more robust safety measures to improve reliability.Source: B Broader impact · p. 15
  • The authors identify offensive content and implicit suggestions of harmful actions as risks of the benchmark scenarios.They report using research-team and AMT review, and truncating potentially harmful model responses to reduce risk.Source: B Broader impact · p. 15
  • The authors judge the risks to be outweighed by the benefits of shedding light on the issue, given their mitigation strategies and the low incremental misuse risk.Source: B Broader impact · p. 15

C Dataset construction

MOSSBench scenarios are generated from principles and examples, then filtered by human evaluation. The appendix also defines refusal-rate evaluation and the criteria used to distinguish refusal from compliance.

C Dataset construction

  • LLMs such as GPT-4 and Gemini-pro 1.5 generate diverse scenario descriptions, which are used to create or retrieve relevant images.The prompts include principles and examples to guide scenario generation.Source: C Dataset construction · p. 15
  • The scenario pipeline has two steps: candidate generation followed by candidate filtering.Generated scenarios are filtered through human evaluation.Source: C Dataset construction · p. 16

D.1 Evaluations

  • Oversensitivity is measured by refusal rates; refusal means not directly answering the image-context question, while compliance means attempting an answer regardless of accuracy.Source: §D.1 · pp. 16, 18

C Dataset construction

  • The Negated Harm prompts pair harmful objects with a contrasting hashtag that signals a positive or safe intention.Source: §D.1 · p. 17

D.1 Evaluations

  • The Counterintuitive Interpretation prompts pair a benign request with a scenario that admits a highly unlikely catastrophic interpretation.Source: §D.1 · p. 17

D.1.1 Human evaluation

Human evaluators apply response guidelines with defined edge cases, and the authors report high agreement. GPT evaluation provides an automated alternative, while experiment settings aim for deterministic outputs.

D.1.1 Human evaluation

  • Human evaluation treats partial refusals as refusals, but irrelevant attempted answers and responses with safety disclaimers as compliance.Source: §D.1.1 · p. 18
  • 97.8% agreement was achieved by the paper's authors when evaluating a random selection of 300 samples.The authors describe this as high inter-annotator agreement.Source: §D.1.1 · p. 18

D.1.2 GPT evaluation

  • For GPT_evaluation, the study uses gpt-4-turbo-2024-04-09, selected for good safety alignment on its samples, to scale evaluation and provide a consistent automated assessment of refusal rates, using a prompt that mirrors human protocols.Source: §D.1.2 · p. 18

D.2 Experimental settings

  • The experiments set model hyperparameters for deterministic output, including temperature 0 across the specified model groups.Gemini and Claude 3 use top-k 1; GPT-4 and open-source models also use random seed 42.Source: §D.2 · p. 18

E.1 Results on contrasting samples

This material reports evaluation and prompt comparisons, then gives qualitative examples of oversensitivity and failures at the perception stage.

E.1 Results on contrasting samples

  • Table 5 documents deterministic zero-shot refusal rates on contrasting samples, while Table 3 details proprietary models.The supplied passages do not include the numerical results.Source: §E.1 · p. 18, Table 3 · p. 19, Table 5 · p. 20

E.2 Mitigation and worsening of oversensitivity

  • Three system prompts target helpfulness, scrutiny, and safety, with refusal-rate results reported in Table 6.The supplied passages do not include the table’s numerical results.Source: §E.2 · p. 19, Table 6 · p. 20

F More qualitative results

F.1 More examples for Claude 3 and Gemini Advanced

  • Examples show Claude 3 exaggerating marching-related risk, while Claude 3 and Gemini-Pro 1.5 misread an objection to animal entertainment.Another example shows Claude 3 interpreting a request as illegal littering despite no image evidence of illegal activity.Source: Figure 16 · p. 21, Figure 17 · p. 22, Figure 18 · p. 22

F.2 Response process examples

  • Different stimuli tend to produce errors at perception, intent reasoning, or safety decision-making stages.Source: §F.2 · p. 22

F.2.1 Perception

F.2.2 Intent reasoning

Examples show that oracle descriptions can expose or correct perceptual errors, while some oversensitive safety judgments persist even with oracle information. The section also illustrates cognitive-distortion-like refusals and describes AMT participant protections.

F.2.2 Intent reasoning

  • Oracle descriptions help in some intent-reasoning examples, including one where the model shifts from refusing a full song to providing a short verse.In another example, the description reveals a perceptual error that the model corrects when conditioned on it.Source: Figure 23 · p. 26, §F.2.2 · p. 26, Figure 24 · p. 27

F.2.3 Safety judgement

  • Oracle descriptions and intent can mitigate oversensitivity in some cases, but other examples remain unresolved even with oracle information.Oracle intent helps in examples involving peaceful protest and opposition to hate against minorities, while another model refuses despite understanding the intent.Source: §F.2.3 · p. 29, Figure 26 · p. 30, Figure 27 · p. 31

F.3 Refusal responses

  • Additional refusal examples exhibit mind reading, catastrophizing, and mental filtering.Source: §F.3.1 · p. 34

F.3.1 Cognitive Distortions

F.3.2 Moralizing

  • Models sometimes moralize by refusing requests as unethical while making unrealistic criticisms or applying judgments inconsistently.Examples include refusals to advise on trapping fireflies, cooking chicken, or splitting cookies despite compliance with another request.Source: §F.3.2 · p. 40, Figure 37 · p. 41, Figure 38 · p. 42

G Crowdsourcing

Loading 2406.17806v1…