PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Situational Safety

Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Anderson Compalas, Dawn Song, Xin Eric Wang

arXiv:2410.06172v2cs.AIcs.CL

TL;DR

The paper addresses whether MLLMs can judge a language query’s safety from its visual context, a gap in assessments focused on clearly unsafe language. It builds MSSBench and analyzes model performance, finding that current models struggle with unsafe situations and that multi-agent pipelines improve average safety accuracy, though performance remains imperfect.

  • Problem

    Existing MLLM safety assessments largely focus on clearly unsafe language, while assistant scenarios can require judging query safety from the visual context.

    Source: §1 · p. 2

  • Method

    The paper introduces MSSBench, a balanced set of chat and embodied language-image scenarios, and evaluates MLLMs with variants examining explicit safety reasoning, visual understanding, and situational safety reasoning.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 6

  • Results

    Current MLLMs struggle to recognize unsafe situations, especially in embodied scenarios, while multi-agent pipelines improve average safety accuracy for almost all tested models but remain imperfect.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings identify visual understanding and safety judgment as continuing challenges for multimodal situational safety.

    Source: §6 · p. 10

    Multi-agent pipelines remain imperfect, particularly because of MLLMs’ visual-understanding and safety-judgment limitations, and take longer to answer queries.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are rapidly evolving, demonstrating impressive capabilities as multimodal assistants that interact with both humans and their environments. However, this increased sophistication introduces significant safety concerns. In this paper, we present the first evaluation and analysis of a novel safety challenge termed Multimodal Situational Safety, which explores how safety considerations vary based on the specific situation in which the user or agent is engaged. We argue that for an MLLM to respond safely, whether through language or action, it often needs to assess the safety implications of a language query within its corresponding visual context. To evaluate this capability, we develop the Multimodal Situational Safety benchmark (MSSBench) to assess the situational safety performance of current MLLMs. The dataset comprises 1,820 language query-image pairs, half of which the image context is safe, and the other half is unsafe. We also develop an evaluation framework that analyzes key safety aspects, including explicit safety reasoning, visual understanding, and, crucially, situational safety reasoning. Our findings reveal that current MLLMs struggle with this nuanced safety problem in the instruction-following setting and struggle to tackle these situational safety challenges all at once, highlighting a key area for future research. Furthermore, we develop multi-agent pipelines to coordinately solve safety challenges, which shows consistent improvement in safety over the original MLLM response. Code and data: mssbench.github.io.

1 INTRODUCTION

Multimodal situational safety asks whether a query is safe in its visual context, a consideration not captured by assessments centered on clearly unsafe language. The paper introduces a benchmark and evaluation of this challenge, finding that current MLLMs struggle and that multi-agent reasoning improves average safety accuracy while leaving performance imperfect.

1 INTRODUCTION

  • Multimodal situational safety requires judging a language query’s safety from its real-time visual context, rather than from the query alone.A running query may be benign on a clean walkway but dangerous near a cliff.Source: §1 · p. 2
  • MSSBench contains 1,960 language-image pairs with safe and unsafe contexts and covers chat and embodied assistant scenarios.Chat examples use generated activities and queries with LLM and domain-expert filtering; embodied examples use manually created household tasks and simulator-collected visual contexts.Source: §1 · p. 2
  • Current MLLMs struggle to recognize unsafe situations; explicit safety reasoning improves average performance but can cause over-sensitivity in safe situations.Models also perform poorly in embodied scenarios, where precise visual understanding and situational safety judgment are lacking; performance decreases under more-subtask settings due to task complexity.Source: §1 · p. 2
  • Multi-agent situational reasoning pipelines assign safety and query-response subtasks to different agents, improving average safety accuracy for almost all MLLMs while leaving performance imperfect, especially in embodied tasks.The authors report that each subtask can thereby be executed with higher accuracy.Source: §1 · p. 2
  • The evaluation diagnoses explicit safety reasoning, visual understanding, and situational safety reasoning as key aspects of MLLM safety performance.The paper evaluates open-source and proprietary MLLMs on the benchmark and uses variants to examine these aspects.Source: §1 · p. 2

2 RELATED WORK

MLLMs have developed into chat and embodied assistants, but their safety in these application scenarios has not been thoroughly studied. Earlier safety assessments largely focus on clearly unsafe language, whereas this paper frames safety as dependent on visual situation.

2 RELATED WORK

  • MLLMs serve as chat and embodied assistants, while their safety in these application scenarios has not been thoroughly studied.The related-work discussion describes visual question answering and embodied decision-making or task planning as assistant uses.Source: §2 · p. 3
  • Most existing MLLM safety benchmarks target clearly unsafe language and use images as attacks or safety-unrelated context.Examples include query-relevant images, direct text embedding, and optimized adversarial images [Liu et al., 2023c; Gong et al., 2023; Shayegani et al., 2023].Source: §2 · p. 3
  • This work distinguishes multimodal situational safety by treating query safety as variable across visual situations and evaluating it in chat and embodied scenarios.The related-work passage also notes concurrent studies of over-sensitivity in MLLMs [Li et al., 2024c; Wang et al., 2024b].Source: §2 · p. 3

3 MULTIMODAL SITUATIONAL SAFETY

MSSBench evaluates query safety in its visual context across chat and embodied scenarios, using category-based data generation, filtering, and human verification. The benchmark is balanced between safe and unsafe situations, while its reported results show models struggle particularly in embodied tasks.

3.1 DATASET OVERVIEW

  • MSSBench scores query intent in real-time visual context; items pair queries with safe or unsafe scenes across chat and embodied tasks, four domains, and ten categories.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 CHAT DATA COLLECTION

  • The chat pipeline generates activities and unsafe situations, filters category mismatches and impossible activities, retrieves unsafe images, then human-validates pairs before query generation.Source: §3.2 · pp. 4–5

3.3 EMBODIED DATA COLLECTION

  • Five embodied task categories yield 38 safe and 38 unsafe tasks and 380 templated instructions; collected scenario images bring the subset to 760 samples.Source: §3.3 · p. 5
  • All evaluated MLLMs struggle to respond safely under unsafe situations and perform even worse on embodied tasks.Source: Table 2 · p. 6

3.4 DATA STATISTICS

  • The benchmark contains 1,960 pairs: 1,200 chat and 760 embodied, with half of all situations safe and half unsafe.Source: Table 1 · p. 4, §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate eight open-source and proprietary MLLMs and find that they struggle to recognize unsafe situations, particularly in embodied scenarios. Diagnosis tests explicit safety reasoning, intent classification, and visual understanding as factors in performance.

4.1 SETUP

  • The evaluation covers five open-source and three proprietary MLLMs, with GPT-4o categorizing instruction-following responses as safe or unsafe for accuracy scoring.The open-source models were evaluated in their 7B versions using default settings.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • Claude 3.5 Sonnet achieves the highest average accuracy at 64.0%, while all evaluated models perform worse in embodied scenarios, especially for unsafe situations.Proprietary models perform better in unsafe scenarios without being over-sensitive in safe ones, yielding higher average accuracy than open-source models.Source: §4.2 · p. 6

4.3 RESULT DIAGNOSIS

  • Explicitly classifying query or intent safety improves performance, particularly in unsafe situations, but reduces accuracy in safe situations and makes models more over-sensitive.The evaluation variants also test intent classification with self-generated or ground-truth situation captions.Source: §4.3 · p. 7, Figure 4 · p. 7
  • In embodied tasks, even proprietary models achieve only around 59% accuracy; errors include weak safety knowledge, missed or misidentified visual details, and ignored task-relevant factors.The authors attribute some visual errors to less training on embodied data and subtler safety-relevant visual factors.Source: §4.3 · pp. 7–8
  • Classifying intent is more accurate than classifying the query for both open- and closed-source models, with query classification adding the task of recognizing potential intent.Reported query-classification errors include overlooking unsafe context, hallucinating safety concerns, and not following the requested judgment task.Source: §4.3 · p. 8
  • Ground-truth captions improve both open-source and proprietary models, with larger gains for open-source models; visual understanding is not a significant bottleneck for proprietary models in chat.The caption is manually written to faithfully describe the image and include information needed for safety judgment.Source: §4.3 · p. 8
  • Self-captioning improves open-source chat performance but decreases performance for both model groups in embodied scenarios, where irrelevant caption details can mislead safety judgments.The authors suggest weaker vision-language alignment may explain why open-source models sometimes identify relevant image details in captions but ignore them in direct safety judgments.Source: §4.3 · p. 8

5 MULTI-AGENT SYSTEM FOR BETTER SAFETY REASONING

The paper divides safety answering into agent-specific subtasks, with different designs for chat and embodied scenarios. The multi-agent pipeline improves performance for almost all models, though embodied-task performance remains imperfect.

5.1 MULTI-AGENT SYSTEM DESIGN

  • For chat, open-source models use intent-reasoning, visual-understanding, safety-judgment, and question-answering agents; proprietary models omit the visual-understanding agent.For embodied tasks, one agent locates the important environment state and another judges task safety and generates a response focused on that state.Source: §5.1 · p. 9

5.2 RESULT AND ANALYSIS

  • In embodied tasks, the multi-agent design improves proprietary models more, while most open-source models show no significant improvement; replacing the reasoned environment state with ground truth improves both tested models.The replacements indicate errors in locating the important state, visual recognition, or safety judgment; even GPT-4o remains far from perfect.Source: §5.2 · pp. 9–10
  • The multi-agent pipeline improves performance for almost all models across chat and embodied subtasks, and helps open-source chat models reach Gemini-level performance.Most open-source models do not improve in the one-step reasoning baseline.Source: §5.2 · p. 9, Figure 7 · p. 10

6 CONCLUSION AND LIMITATIONS

The paper concludes that MLLMs struggle to recognize unsafe situations, especially in embodied scenarios, while explicit safety reasoning and better safety-relevant visual understanding can improve performance. Multi-agent approaches improve safety, but performance remains imperfect and responses take longer.

6 CONCLUSION AND LIMITATIONS

  • Table 3 compares the embodied multi-agent framework across Multi-Agent, GT Environment State, and GT Observation settings.Source: Table 3 · p. 10
  • Current MLLMs face significant challenges recognizing unsafe situations, particularly in embodied scenarios; explicit safety reasoning and better safety-relevant visual understanding can improve performance.The conclusion identifies these as findings from its diagnostic analysis.Source: §6 · p. 10
  • Multi-agent approaches assign different subtasks to different agents to improve safety performance, but visual-understanding and safety-judgment limitations leave results imperfect.The pipelines also take longer to answer because the model reasons through multiple steps.Source: §6 · p. 10

A.1 PER CATEGORY PERFORMANCE IN INSTRUCTION FOLLOWING SETTING FOR CHAT TASK

In the chat instruction-following setting, open-source models perform steadily in safe scenarios but struggle in unsafe ones; closed-source models also do well in safe scenarios, with category-specific weaknesses in unsafe scenarios.

A.1 PER CATEGORY PERFORMANCE IN INSTRUCTION FOLLOWING SETTING FOR CHAT TASK

  • Open-source models’ accuracy in unsafe chat scenarios is often below 10%, especially for illegal activities and offensive behavior.Their safe-scenario performance is stable.Source: §A.1 · p. 14, Figure 8 · p. 14
  • Closed-source models perform well in safe chat scenarios, but illegal activity is their weakest unsafe category.Source: §A.1 · p. 14, Figure 9 · p. 14
  • For closed-source models, unsafe-scenario performance on offensive behavior is weaker than on property damage and physical harm.Source: §A.1 · p. 14, Figure 9 · p. 14

A.2 PERFORMANCE OF MLLMS IN MULTIMODAL SITUATIONAL SAFETY UNDER INTENT BINARY SAFETY CLASSIFICATION SETTING FOR CHAT TASK

Under binary safety classification for chat, open-source models are stable across safe categories but decline in unsafe situations; closed-source models show the reverse pattern, with stronger overall performance in unsafe contexts.

A.2 PERFORMANCE OF MLLMS IN MULTIMODAL SITUATIONAL SAFETY UNDER INTENT BINARY SAFETY CLASSIFICATION SETTING FOR CHAT TASK

  • Open-source models are stable across four safe chat categories, particularly for illegal activities, but performance declines in unsafe situations.Offensive behaviors and illegal activities are relatively stronger unsafe categories; property damage and physical harm are more complex and subtle.Source: §A.2 · p. 15, Figure 10 · p. 15
  • The binary classification comparison concerns model performance by user intent in safe and unsafe chat situations.Source: Figure 10 · p. 15
  • Closed-source models show stable performance across safe chat categories, but their performance is lower than in unsafe contexts, where overall results are stronger.The authors attribute the safe-context pattern to over-sensitivity and higher misjudgment rates; unsafe-category results are comparable, with property damage slightly better.Source: §A.2 · p. 15

A.3 PERFORMANCE OF MLLMS IN MULTIMODAL SITUATIONAL SAFETY UNDER INTENT BINARY SAFETY CLASSIFICATION SETTING FOR EMBODIED TASK

In embodied tasks, open-source models perform strongly in safe situations but fall below 40% in unsafe ones; closed-source models are weaker in safe situations and exceed 80% at their peak in unsafe scenarios.

A.3 PERFORMANCE OF MLLMS IN MULTIMODAL SITUATIONAL SAFETY UNDER INTENT BINARY SAFETY CLASSIFICATION SETTING FOR EMBODIED TASK

  • Open-source models achieve nearly 100% accuracy in safe physical tasks, but scores for both embodied tasks fall below 40% in unsafe situations.Source: §A.3 · p. 15, Figure 12 · p. 16
  • Closed-source models perform more weakly in safe embodied scenarios than unsafe ones, with peak accuracy above 80% across both categories in unsafe scenarios.The two categories perform similarly in safe scenarios.Source: §A.3 · p. 15, Figure 12 · p. 16
  • The embodied binary-classification figures compare open- and closed-source models across safe and unsafe situations.Source: Figure 12 · p. 16

A.4 EVALUATION

The evaluation compares response-label distributions across safe and unsafe scenarios under multi-agent and baseline settings, and categorizes chat and embodied responses into four safety levels.

A.4 EVALUATION

  • The evaluation tables categorize MLLM responses into four safety levels for chat and embodied tasks.Source: Table 4 · p. 17, Table 5 · p. 17
  • The multi-agent framework shifts predictions toward labels 0 and 1, mitigating baseline models’ tendency to favor label 2 across safe and unsafe scenarios.Closed-source models more often give clear warnings in unsafe scenarios, but can also show excessive safety sensitivity.Source: §A.4 · pp. 16–17, Figure 13 · p. 18

A.5 RESULT DIAGNOSIS

Case studies illustrate how multi-agent reasoning handles unsafe and safe situations in chat and embodied tasks. The section also outlines diagnosis settings, agent prompts, and examples used in data collection.

A.5 RESULT DIAGNOSIS

  • Diagnosis compares four binary safety-classification bases—query, intent, intent with self-generated captions, and intent with ground-truth activity captions—alongside multi-agent intent classification and instruction following against a single-step baseline.Source: Table 6 · p. 19, §A.5 · pp. 19–20, Figure 14 · p. 19

A.6 CASE STUDY

A.7 PROMPT

  • Embodied prompts ask for safety judgments or action plans, with a key-object locator identifying environmental items that may affect injury or property-damage risk.Source: Figure 15 · p. 20, §A.7 · pp. 24–25
  • Chat prompts separately elicit intent and image-based safety judgments, while another prompt asks for an image description and a combined prompt judges intent safety in context.Source: §A.7 · pp. 24, 26
  • Chat response prompts branch to answering safe queries, warning about unsafe intent, or deciding whether to answer or warn; a baseline prompt answers directly.Source: §A.7 · pp. 24, 26

A.8 MORE EXAMPLES

Related papers

Loading 2410.06172v2…