PaperlayerView exact revision on arXiv

Source-linked AI summary

How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs

Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, Cihang Xie

arXiv:2311.16101v1cs.CVcs.CLcs.LG

TL;DR

Existing VLLM safety evaluations have covered limited tasks or input modalities. This paper introduces a benchmark spanning OOD scenarios and red-teaming attacks, finding that VLLMs handle OOD visual content better than perturbed OOD text.

  • Problem

    Prior VLLM safety evaluations covered limited tasks or input modalities, leaving a need for a broader safety assessment.

    Source: §1 · pp. 1–2

  • Method

    The benchmark evaluates VLLMs on OOD scenarios and red-teaming attacks targeting visual and language components.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    VLLMs excel at OOD visual content but struggle with counterfactual OOD text, while sketch images challenge even GPT-4V.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings underscore the need to incorporate safety protocols during visual instruction tuning.

    Source: §1 · p. 2

Abstract

from arXiv · show

This work focuses on the potential of Vision LLMs (VLLMs) in visual reasoning. Different from prior studies, we shift our focus from evaluating standard performance to introducing a comprehensive safety evaluation suite, covering both out-of-distribution (OOD) generalization and adversarial robustness. For the OOD evaluation, we present two novel VQA datasets, each with one variant, designed to test model performance under challenging conditions. In exploring adversarial robustness, we propose a straightforward attack strategy for misleading VLLMs to produce visual-unrelated responses. Moreover, we assess the efficacy of two jailbreaking strategies, targeting either the vision or language component of VLLMs. Our evaluation of 21 diverse models, ranging from open-source VLLMs to GPT-4V, yields interesting observations: 1) Current VLLMs struggle with OOD texts but not images, unless the visual information is limited; and 2) These VLLMs can be easily misled by deceiving vision encoders only, and their vision-language training often compromise safety protocols. We release this safety evaluation suite at https://github.com/UCSC-VLAA/vllm-safety-benchmark.

1. Introduction

The introduction identifies a gap in VLLM safety evaluation and proposes a suite spanning out-of-distribution scenarios and attacks on both visual and language components.

Introduction

  • OODCV-VQA and Sketchy-VQA use rarely seen images and variants with counterfactual descriptions or less-common main objects.The datasets are based on existing image corpora [57]; counterfactual descriptions follow Zhang et al. [55].Source: §1 · p. 2
  • The benchmark also evaluates white-box and transfer jailbreaking attacks on vision and language inputs [32] [63].The authors evaluate 20 open-source VLLMs across model scales, LLM versions, and vision encoders, plus GPT-4V on selected challenging cases.Source: §1 · p. 2
  • VLLMs handle OOD visual content across texture, weather, pose, and shape, but struggle when language input is perturbed counterfactually.The contrast highlights the role of language inputs in VLLM functionality.Source: §1 · p. 2
  • VLLMs, including GPT-4V, struggle with sketch images, finding even simple yes/no questions challenging because sketches provide limited information.Source: §1 · p. 2
  • Simple CLIP ViT-based attacks can mislead VLLMs that fail to reject irrelevant inputs, but GPT-4V is less susceptible and can refuse inappropriate inputs.The proposed vision-encoder attack aligns CLIP ViT with irrelevant textual objects and performs comparably to or better than the latest ensemble-based attack [16].Source: §1 · p. 2
  • Attacking only the vision input can mislead VLLMs into irrelevant responses, but eliciting specific toxic responses through vision-only jailbreaking is challenging.Source: §1 · p. 2
  • Vision-language training weakens aligned language models’ safety protocols in most cases, motivating safety rules during visual instruction tuning.Source: §1 · p. 2

2. The Safety Evaluation Benchmark

The benchmark evaluates out-of-distribution generalization and redteaming attacks across recent open-source VLLMs and GPT-4V [31]. It also aims to release its datasets and code for future research.

2.1.1 OODCV-VQA and its Counterfactual Variant

  • OODCV-VQA uses OODCV images [57] and templated yes/no or digit questions, while its counterfactual variant appends descriptions that alter answers.For yes/no questions, answers are flipped; digit questions alter item counts or remove irrelevant objects.Source: §2.1.1 · pp. 2–3

2.1.2 Sketchy-VQA and its Challenging Variant

  • Sketchy-VQA contains 2,000 images and 4,000 yes/no instances across 50 frequent object classes, while Sketchy-Challenging uses the 50 least frequent category names.The dataset uses sketch images from, selecting 40 images per class for the frequent-category set [38].Source: §2.1.2 · p. 3

2.2.1 Misleading VLLM Outputs by Attacking Off-the-Shelf ViT only

  • The misleading attack trains image noise to match CLIP representations with irrelevant text, using single-phrase or mixed-text variants and a 200-image annotated benchmark.The images come from NIPS17 and are cross-validated by two annotators; the attack maximizes image-text similarity using CLIP encoders [33] [40].Source: §2.2.1 · p. 3

2.2.2 Jailbreaking VLLMs to Generate Toxic Outputs

  • The vision-encoder jailbreak evaluation tests white-box and transfer attacks using adversarial images trained on MiniGPT4, LLaVA, and InstructBLIP, scoring toxicity with Perspective API.The visual attacks follow prior training and testing settings [21] [32], maximize harmful-output probability, and deem responses toxic above a threshold such as 0.5/1.0.Source: §2.2.2 · p. 4
  • LLM-part attacks compare vanilla, white-box, and transfer settings on 100 toxic behaviors, using GPT-3.5 to classify harmful outputs and testing transfer strings on GPT-4V [30].The attacks use an automatic LLM attack system [53] [63]; transfer strings from both LLMs and VLLMs are applied to GPT-4V as an oracle LLM.Source: §2.2.2 · p. 4

3. Vision Large Language Models

The evaluation covers 20 open-source VLLMs across 10 modeling categories and also tests GPT-4V on challenging OOD and misleading-attack examples. The selected models and verification procedures define the evaluation scope.

3. Vision Large Language Models

  • The open-source evaluation includes 20 models across 10 categories, each using a 7B or 13B LLM and typically undergoing visual pre-training and visual instruction tuning.These stages aim to support basic visual understanding and alignment with human preferences.Source: §3 · p. 4
  • GPT-4V is tested on 650 challenging examples across four OOD datasets and misleading-attack data, with human annotators examining its justifications.The examples were selected because all four InstructBLIP models answered them incorrectly.Source: §3 · p. 4

4. Evaluation Results

VLLMs handle simple visual questions about out-of-distribution images better than counting, counterfactual instructions, or sketch recognition. They can also be misled by attacks on vision encoders, while visual jailbreaking transfers poorly and vision-language tuning weakens safety protocols.

4.1.1 OODCV-VQA and its Counterfactual Variant

  • VLLMs generally exceed 95% accuracy on OODCV Yes/No appearance questions, but struggle with counting; digit-answer accuracy falls at least 5% versus VQAv2 [55].InstructBLIP and LLaVA v1.5, Qwen-VL-Chat, CogVLM, and InternLM-X each achieve over 70% average overall accuracy on the OODCV benchmark.Source: §4.1.1 · pp. 4–5, Table 5 · p. 5
  • Counterfactual suffixes reduce mean overall accuracy by 17.1% and Yes/No accuracy by 33.2%; GPT-4V drops 11.6% overall and 14.1% on counting.PandaGPT and Fuyu average only 2% accuracy on counterfactual Yes/No questions, and case studies show models hallucinating despite answering the original question correctly.Source: Table 5 · p. 5, §4.1.1 · pp. 5–6

4.1.2 Sketchy-VQA and its Challenging Variant

  • Sketchy-VQA remains difficult: unfamiliar sketch objects lower average F1 by 4.4%, the best F1 stays below 70%, and GPT-4V excels over open VLLMs.Models with more powerful LLM components generally perform better on these challenging tasks.Source: Figure 2 · p. 5, §4.1.2 · p. 6

4.2.1 Misleading Attack through the ViT

  • Attacks tuned on CLIP-ViT-L-14 misguide PandaGPT, InternLM-X, and Fuyu, but CogVLM’s misleading rate is 34.3% below average; GPT-4V often refuses adversarial-image questions.For GPT-4V, a larger MIX.ATTACK perturbation budget raises both rejection ratio and misleading rate.Source: §4.2.1 · p. 6
  • At 𝜖0 = 64∕255, SIN.ATTACK and MIX.ATTACK improve average misleading rate over ATTACKBARD by 5.0% and 8.4%, respectively; narrow-budget MIX.ATTACK loses 3.6% missing rate.MIX.ATTACK’s stronger performance than SIN.ATTACK is attributed to its more diverse word embeddings for aligning image perturbations using CLIP.Source: §4.2.1 · p. 6, Table 6 · p. 7

4.2.2 Jailbreaking VLLMs

  • Direct visual jailbreak attacks on three targeted models produce toxic outputs 2.1× as often as clean images, but the increase is only about 5% across VLLMs.The authors conclude that visual jailbreaking strategies lack sufficient transferability and robustness for a universal attack.Source: Figure 4 · p. 7, §4.2.2 · p. 7
  • Vision-language tuning weakens LLM safety: VLLMs show 5.5% higher vanilla and 17.3% higher white-box ASRs, while LLM-to-VLLM transfers yield 26.0% higher ASR than reverse.Transferring prompts from LLMs to GPT-4V is also easier than from VLLMs, with white-box ASR improved by 2.7% on average.Source: §4.2.2 · pp. 7–8, Table 7 · p. 8

5. Related Work

VLLMs combine LLMs with visual inputs for visual understanding and conversation, but face safety challenges. Prior evaluations examined visual or language vulnerabilities separately, leaving VLLM safety issues without systematic evaluation.

Vision Large Language Models

  • VLLMs integrate LLMs with visual inputs and show visual understanding and conversational abilities, while remaining vulnerable to adversarial attacks, hallucination, and out-of-distribution problems.They typically jointly decode visual and text tokens or use external multimodal tools for complex tasks.Source: §5 · p. 8

Safety Evaluations

  • Prior work had not systematically evaluated VLLM safety, focusing instead on adversarial vulnerabilities in either visual inputs or language components.These studies evaluated attack-method effectiveness in various styles.Source: §5 · p. 8

6. Discussions

The discussion links VLLM performance and safety to vision-encoder design, tuning configuration, training-data diversity, and the inclusion of safety-focused data. It highlights a tradeoff between OOD recognition and adversarial resistance, alongside weaknesses in current vision-language safety training.

Discussions

  • Fuyu matches leading VLLMs on traditional benchmarks but incurs 18.4% and 4.9% drops on two OOD tasks while achieving 33.5% lower ASR than other VLLMs.Without an explicit pre-trained vision encoder, Fuyu is less vulnerable to adversarial attacks but may struggle to recognize visual content in challenging OOD scenarios.Source: §6 · p. 8
  • With full-parameter tuning, stronger LLMs yield average OOD gains of 52.3% for LLaVA and 10.0% for mPLUG-Owl, whereas MiniGPT4 and InstructBLIP decline 2.1% and 0.2%, respectively.The authors attribute this difference to greater hyperparameter sensitivity when fewer parameters are tuned, calling for careful training-configuration selection.Source: §6 · p. 8
  • InstructBLIP’s 13 diverse, annotated vision-language datasets contrast with MiniGPT4’s 3,500 visual-instruction instances, a data difference the authors link to their performance disparity.Although both models rely on the QFormer, InstructBLIP consistently outperforms most open-source alternatives while MiniGPT4 performs poorly.Source: §6 · p. 8
  • Because vision-language training can undermine LLM safety and current datasets largely lack safety-focused data, the authors call for VLLM-specific safeguards and safer textual instructions.They cite language bias in VLLMs and the need to address hazards such as failing to reject questions prompted by adversarial images.Source: §6 · pp. 8–9

7. Conclusion

The study evaluates VLLM safety across out-of-distribution scenarios and redteaming attacks, assessing 21 models and discussing findings that underscore the need for further safety research.

7. Conclusion

  • The benchmark evaluates VLLMs on two safety dimensions: out-of-distribution scenarios and redteaming attacks.Source: §7 · p. 9
  • The evaluation covers 21 models, including GPT-4V and recent open-source models.Source: §7 · p. 9
  • The authors present five findings and discuss contributing factors, underscoring the need for future research to enhance VLLM safety.Source: §7 · p. 9

Appendices … B. Details of Redteaming Attack

The appendices provide supplementary OOD dataset and evaluation details, examples of VLLM errors, and a preview of the red-teaming attack configurations. They note that template-based counterfactual descriptions can challenge VLLMs and warn readers that the appendix includes unsafe model responses.

Appendices

  • The appendix warns that the material below may contain unsafe model responses.Source: Appendices · p. 12

A. Details of Out-of-Distribution Scenarios

A.1. OODCV-VQA and its Counterfactual Variant

  • Template-based counterfactual descriptions can pose a barrier to current VLLMs answering OOD questions correctly.The paper contrasts this approach with prior work that uses human annotators to examine questions after counterfactual phrases are added.Source: §A.1 · p. 12
  • For OODCV-VQA and its counterfactual variant, the appendix provides full question templates in Table 8.GPT4V questions receive an added request for a short, concise response with one numerical or boolean answer to avoid long justifications.Source: §A.1 · p. 12
  • Examples show errors on unusual images and complex counterfactual questions, including wrong motorbike counts, hallucinated unicorns, and confusing car answers.The examples report wrong motorbike counts from most VLLMs, including GPT4V; MiniGPT4 and LLaVAv1.5 hallucinate unicorns and remain wrong with counterfactual descriptions, while MiniGPT4, LLaVAv1.5, and CogVLM err on complex counterfactual questions. Figure 5 illustrates an OODCV-VQA question and its counterfactual version.Source: §A.1 · p. 12, Figure 5 · p. 12

A.2. Sketchy-VQA and its Challenging Variant

  • For Sketchy-VQA and its challenging variant, the appendix provides 50 filtered image labels and full question templates.GPT4V questions also receive the instruction to keep responses short and concise.Source: §A.2 · p. 12

B. Details of Redteaming Attack

B.1. Misleading Attack

The section evaluates misleading attacks using annotated images and reports that most VLLMs, including GPT-4V, struggle to resist the attack, while CogVLM is relatively more reliable. The comparison with ATTACK-BARD excludes MiniGPT4 and InstructBLIP because BARD was ensemble-trained on those models.

B.1. Misleading Attack

  • The evaluation uses 200 NIPS17 images with prominent scene objects labeled by two human annotators, testing Gaussian noise and attack-generated noise.For each image, an attack succeeds only if the model gives label-irrelevant responses under all four instructions.Source: §B.1 · pp. 12–13
  • SIN.ATTACK uses dog, spaceship, and coconut as irrelevant misleading concepts, while MIXATTACK trains noise using CLIP text- and image-embedding matching.The study uses official ATTACK-BARD attacked images and reports SIN.ATTACK’s average misleading rate across its three attacked-image sets.Source: §B.1 · p. 13
  • ATTACK-BARD comparisons exclude MiniGPT4 and InstructBLIP because BARD was ensemble-trained on those models, while the study focuses on transfer attacks.The authors therefore do not report misleading results for the MiniGPT4 and InstructBLIP model families.Source: §B.1 · p. 13
  • Most VLLMs, including GPT-4V, struggle to defend against the attack; CogVLM has a significantly lower misleading rate and is relatively reliable against it.In the examples, most open-source models mistake a hummingbird for a person, GPT-4V calls it an insect, and all models except CogVLM are misled by a chandelier image.Source: §B.1 · pp. 13–14

B.2. Jailbreak VLLMs through ViTs

Adversarially trained images can jailbreak VLLMs into producing toxic outputs, underscoring the need for safer and more reliable vision-language systems.

B.2. Jailbreak VLLMs through ViTs

  • The attack trains adversarial examples for MiniGPT4, LLaVA, and InstructBLIP using 5,000 PGD iterations with batch size 8 on a 66-sentence harmful corpus.The corpus targets gender, race, and humanity as a whole; its sentences were produced using an early LLaMA model lacking safety alignment features.Source: §B.2 · p. 14
  • Adversarially trained images jailbreak VLLMs: two examples show toxic outputs from two different models when prompted to complete a sentence.The models receive adversarial images and the prompt “Please extend the following sentence:”.Source: §B.2 · p. 14

B.3. Jailbreak VLLMs through LLMs

The section shows that adversarial suffixes trained on either Vicuna v1.5 or LLaVA v1.5 can jailbreak both models. It also specifies how the LLM components were trained and how GPT-4 transfer attacks were evaluated.

B.3. Jailbreak VLLMs through LLMs

  • Training followed Zou et al.’s procedure [63], using batch size 512 on an 80G A100 for 1,000 steps on only each VLLM’s LLM component.Only the LLM part of each VLLM was trained and tested.Source: §B.3 · p. 15
  • For transfer attacks on GPT-4, GPT-4-32k-0613 generated responses, and GPT-3.5-turbo-16k-0613 judged whether they were harmful or hazardous.Source: §B.3 · p. 15
  • Adversarial suffixes trained from either Vicuna v1.5 or LLaVA v1.5 jailbreak both Vicuna v1.5 and LLaVA v1.5.The examples show unsafe responses for both models under either suffix source.Source: §B.3 · p. 15, Figure 12 · p. 17, Figure 13 · p. 17
Loading 2311.16101v1…