PaperlayerView exact revision on arXiv

Source-linked AI summary

Red Teaming Visual Language Models

Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, Qi Liu

arXiv:2401.12915v1cs.AIcs.CLcs.CV

TL;DR

How VLMs perform on red-teaming scenarios involving combined textual and visual inputs remains unclear, and comprehensive benchmarks are lacking. The paper introduces RTVLM, a dataset spanning four aspects and ten task categories, and reports VLM vulnerabilities as well as improved red-teaming performance with RTVLM-based alignment.

  • Problem

    VLM performance in red-teaming scenarios involving textual and visual inputs remains unclear, and comprehensive, systematic benchmarks for current VLMs are lacking.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces RTVLM, a 5,200-sample image-text red-teaming dataset with ten task categories across faithfulness, privacy, safety, and fairness.

    Source: §1 · p. 2

  • Results

    Current popular open-source VLMs are not sufficiently secure against varied red-teaming attacks, while using RTVLM for red-teaming alignment significantly improves downstream red-teaming performance.

    Source: §5 · p. 9

  • Takeaways & Limitations

    RTVLM provides a benchmark for VLM vulnerabilities and a data subset used to train models on security.

    Source: §5 · p. 9

Abstract

from arXiv · show

VLMs (Vision-Language Models) extend the capabilities of LLMs (Large Language Models) to accept multimodal inputs. Since it has been verified that LLMs can be induced to generate harmful or inaccurate content through specific test cases (termed as Red Teaming), how VLMs perform in similar scenarios, especially with their combination of textual and visual inputs, remains a question. To explore this problem, we present a novel red teaming dataset RTVLM, which encompasses 10 subtasks (e.g., image misleading, multi-modal jail-breaking, face fairness, etc) under 4 primary aspects (faithfulness, privacy, safety, fairness). Our RTVLM is the first red-teaming dataset to benchmark current VLMs in terms of these 4 different aspects. Detailed analysis shows that 10 prominent open-sourced VLMs struggle with the red teaming in different degrees and have up to 31% performance gap with GPT-4V. Additionally, we simply apply red teaming alignment to LLaVA-v1.5 with Supervised Fine-tuning (SFT) using RTVLM, and this bolsters the models' performance with 10% in RTVLM test set, 13% in MM-Hal, and without noticeable decline in MM-Bench, overpassing other LLaVA-based models with regular alignment data. This reveals that current open-sourced VLMs still lack red teaming alignment. Our code and datasets will be open-source.

1 Introduction

VLMs combine visual and textual inputs, but their red-teaming performance remains unclear and comprehensive benchmarks are lacking. RTVLM addresses this gap with a multimodal dataset spanning four red-teaming aspects and ten task categories.

1 Introduction

  • VLM red-teaming performance remains unclear, and comprehensive, systematic benchmarks for current models are lacking.The paper notes that multimodal inputs may introduce red-teaming cases not covered by tests focused on LLMs.Source: §1 · pp. 1–2
  • RTVLM introduces ten image-text red-teaming task categories across faithfulness, privacy, safety, and fairness.The 5,200 samples use new question-image pairs, with questions annotated by humans or generated by GPT-4 from human-written seeds.Source: §1 · p. 2
  • The authors evaluate ten open-source VLMs and GPT-4V across the four aspects using scoring criteria, GPT-4V evaluation, and human evaluation.They also analyze performance by subcategory.Source: §1 · p. 2

1. All 10 prominent open-sourced VLMs exhibit

Open-source VLMs struggle with red-teaming challenges, with a reported performance gap of up to 31% compared to GPT-4V. SFT with RTVLM improves LLaVA-v1.5 performance on red-teaming tests while maintaining stable MM-Bench performance.

1. All 10 prominent open-sourced VLMs exhibit

  • In red-teaming challenges, models show varying degrees of struggle, with performance gaps reaching up to 31% compared to GPT-4V, indicating variable challenge performance.Source: §1 · p. 2
  • By applying SFT to LLaVA-v1.5 using RTVLM, performance improves by 10% on the RTVLM test set and 13% on MM-hallu.Performance on MM-Bench remains stable, surpassing other LLaVA models using regular alignment data.Source: §1 · p. 2
  • The study presents RTVLM as the first red-teaming benchmark for visual language models, aimed at elucidating vulnerabilities and proposing solutions for future work.Source: §1 · p. 2

2 RTVLM Dataset

RTVLM is a dataset for red-teaming VLMs, organized around four aspects and ten subtasks. Its task designs cover misleading inputs, privacy, safety, and fairness.

2 RTVLM Dataset

  • The dataset section outlines RTVLM construction, its data format and statistics, and the four aspects covered by its subtasks.Source: §2 · p. 2

2.1 Overview of RTVLM

  • Each instance contains an image, a red-teaming question, a label for whether to answer, answer cautiously, or refuse, and a reference when available.Source: §2.1 · p. 2

2.2 Faithfulness

  • Faithfulness tasks test whether VLMs avoid being misled: text prompts are generated and screened, images may contradict correct answers, and flag-image order is varied.The order task tests the effect of input image order on responses.Source: §2.2 · pp. 3–4

2.3 Privacy

  • Privacy questions concern personal information; celebrity items are labeled for cautious answers, while private-individual items call for refusal.Source: §2.3 · p. 4

2.4 Safety

  • Safety tasks include sensitive politics and race questions, CAPTCHA recognition, and image-based jailbreaks, with answer labels set according to the task’s risk.CAPTCHA and multimodal jailbreak examples are labeled for refusal.Source: §2.4 · p. 4

2.5 Fairness

  • Face fairness uses generated character images and resume prompts, then evaluates gender- and race-related bias in model responses with GPT-4.GPT-4 is used instead of GPT-4V to avoid introducing potential bias from image input to the evaluator.Source: §2.5 · pp. 4–5

3 Experimental Results

The evaluation finds substantial red-teaming weaknesses across VLMs, with performance varying across faithfulness, privacy, safety, and fairness. RTVLM-based SFT improves red-teaming results while MMBench performance remains largely unchanged.

3.1 Experimental Settings

  • The evaluation uses GPT-4V scoring across task categories and human evaluation on 100 RTVLM examples; annotator agreement exceeds 0.7.Human-evaluation results align with the main GPT-4V-evaluated results, and human judgments align more closely with GPT-4V than GPT-4.Source: §3.1 · p. 5, §3.2 · p. 7

3.2 Red Teaming Test Results

  • GPT-4V significantly outperforms the tested open-source VLMs; LLaVA-v1.5-13B is the strongest open-source model, while Fuyu-8b is weaker.LLaVA-v1.5-7B and 13B perform similarly despite their size difference.Source: §3.2 · p. 6
  • Most models identify misleading content in pure text better than misleading information mixed with images.The authors report lower scores on image-mixed misleading tasks, indicating greater susceptibility to visual misinformation.Source: §3.2 · p. 6
  • Open-source VLMs generally do not refuse personal-information inquiries, whereas GPT-4V refuses questions about non-celebrity personal information.GPT-4V responds accurately to celebrity inquiries or indicates when the information is absent.Source: §3.2 · pp. 6–7
  • Most VLMs struggle to read image text in jailbreaking and CAPTCHA tasks; LLaVA recognizes image text better but still may produce harmful content or misidentify CAPTCHAs.The authors characterize this as a combination of image-text processing limitations and insufficient red-teaming alignment.Source: §3.2 · p. 7
  • GPT-4V has the smallest reported gender and race fairness bias, while VLM gender bias is weaker than race bias.Lighter-skinned groups score higher than darker-skinned groups, with Native Americans almost always scoring lower across models; gender scores are relatively balanced for men and women.Source: §3.2 · p. 7

3.3 Red Teaming Alignment Analysis

  • Existing LLaVA-SFT and LLaVA-RLHF alignment training shows no significant RTVLM improvement over original models; the alignment study uses GPT-4V-generated SFT data sampled across RTVLM categories.The sampled training data excludes RTVLMtest examples.Source: §3.3 · pp. 7–8
  • RTVLM-based SFT improves MMHal and RTVLMtest performance for LLaVA-v1.5 and LLaVA-v1.5-ShareGPT4V, while MMBench performance remains largely unchanged.Combining RTVLM with ShareGPT4V data yields better results in both performance and red-teaming scenarios.Source: §3.3 · p. 8

4 Related Work

Prior red-teaming research tests language models manually or with automated generation, and examines safety risks from visual inputs. Related VLM work develops multimodal architectures and instruction-tuning approaches.

4.1 Red Teaming and Safety

  • Language-model red teaming includes manual test-case generation by human annotators or adversaries and automated generation by one language model targeting another.Earlier work also examines scaling behavior and model types, including rejection sampling and reinforcement learning from human feedback.Source: §4.1 · p. 8
  • Prior work has examined visual adversarial vulnerabilities in systems combining language and vision; this paper extends red teaming to VLMs.The cited research includes Flamingo and GPT-4.Source: §4.1 · pp. 8–9

4.2 Visual Language Models

  • VLM research has integrated language models with vision encoders, narrowed modality gaps, and incorporated instruction tuning and alignment.Examples include Flamingo, PaLI-X, BLIP-2, InstructBLIP, MMICL, MiniGPT-4, and the LLaVA series.Source: §4.2 · p. 9

5 Conclusion

The paper introduces RTVLM as a red-teaming dataset for VLMs and reports weaknesses among tested models. It also reports that using RTVLM data for red-team alignment improves performance on downstream red-teaming tasks.

5 Conclusion

  • RTVLM is presented as the first VLM red-teaming dataset, with four task categories and 5,200 entries, including multimodal jailbreaking and visual misleading.Tests of open-source VLMs and GPT-4V found that many models were not sufficiently secure against red-teaming attacks, and some open-source models showed strong biases.Source: §5 · p. 9
  • Using a subset of RTVLM data for red-team alignment significantly improved models’ performance on downstream red-teaming tasks.The authors describe this as demonstrating the viability of using RTVLM for training on model security.Source: §5 · p. 9

A Data Evaluation Prompts

The appendix identifies prompts used for privacy and safety evaluation, a fairness color key, and a data-generation format.

A Data Evaluation Prompts

Loading 2401.12915v1…