PaperlayerView exact revision on arXiv

Source-linked AI summary

IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselves

Ruofan Wang, Juncheng Li, Yixu Wang, Bo Wang, Xiaosen Wang, Yan Teng, Yingchun Wang, Xingjun Ma, Yu-Gang Jiang

arXiv:2411.00827v6cs.CVcs.AI

TL;DR

VLM jailbreak evaluation is limited by attacks that depend on white-box access or human-engineered image pipelines. IDEATOR uses a VLM and diffusion model to generate multimodal jailbreak pairs, achieving 94% ASR on MiniGPT-4 and transferring across three other VLMs.

  • Problem

    Existing VLM jailbreak methods often require white-box access or human-engineered pipelines, while adversarial images can lack semantic meaning and be detected by safety mechanisms.

    Source: §1 · p. 1

  • Method

    IDEATOR uses a VLM as an adaptive jailbreak agent and a diffusion model to generate multimodal jailbreak prompts for black-box attacks.

    Source: §1 · p. 2

  • Results

    94% ASR on MiniGPT-4 with an average of 5.34 queries, and transfer ASRs of 82% on LLaVA, 88% on InstructBLIP, and 75% on Chameleon.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Evaluations with VLJailbreakBench reveal significant gaps in current VLM safety mechanisms, underscoring the need for stronger defenses.

    Source: §1 · p. 2

    VLJailbreakBench remains relatively small in scale and requires further computational resources and automated selection.

    Source: §5.3 · p. 8

Abstract

from arXiv · show

As large Vision-Language Models (VLMs) gain prominence, ensuring their safe deployment has become critical. Recent studies have explored VLM robustness against jailbreak attacks-techniques that exploit model vulnerabilities to elicit harmful outputs. However, the limited availability of diverse multimodal data has constrained current approaches to rely heavily on adversarial or manually crafted images derived from harmful text datasets, which often lack effectiveness and diversity across different contexts. In this paper, we propose IDEATOR, a novel jailbreak method that autonomously generates malicious image-text pairs for black-box jailbreak attacks. IDEATOR is grounded in the insight that VLMs themselves could serve as powerful red team models for generating multimodal jailbreak prompts. Specifically, IDEATOR leverages a VLM to create targeted jailbreak texts and pairs them with jailbreak images generated by a state-of-the-art diffusion model. Extensive experiments demonstrate IDEATOR's high effectiveness and transferability, achieving a 94% attack success rate (ASR) in jailbreaking MiniGPT-4 with an average of only 5.34 queries, and high ASRs of 82%, 88%, and 75% when transferred to LLaVA, InstructBLIP, and Chameleon, respectively. Building on IDEATOR's strong transferability and automated process, we introduce the VLJailbreakBench, a safety benchmark comprising 3,654 multimodal jailbreak samples. Our benchmark results on 11 recently released VLMs reveal significant gaps in safety alignment. For instance, our challenge set achieves ASRs of 46.31% on GPT-4o and 19.65% on Claude-3.5-Sonnet, underscoring the urgent need for stronger defenses. VLJailbreakBench is publicly available at https://roywang021.github.io/VLJailbreakBench.

1. Introduction

VLM jailbreak evaluation is constrained by attacks that require white-box access or human-engineered pipelines, while multimodal integration creates new safety vulnerabilities. IDEATOR uses a VLM and diffusion model to generate jailbreak pairs, and supports a 3,654-sample benchmark of 11 VLMs.

  • Combining textual and visual modalities introduces vulnerabilities, and malicious prompts can elicit harmful outputs otherwise restricted by VLM safeguards.Source: §1 · p. 1
  • Prior methods use white-box access or human-engineered image-generation pipelines, limiting real-world applicability, flexibility, and scalability.Source: §1 · pp. 1–2
  • IDEATOR combines a VLM jailbreak agent with a diffusion model to automatically generate effective, transferable, and diverse image-text jailbreak pairs.Source: §1 · p. 2
  • 94% ASR on MiniGPT-4 with an average of 5.34 queries, with transfer ASRs of 82% on LLaVA, 88% on InstructBLIP, and 75% on Chameleon.Source: §1 · p. 2
  • VLJailbreakBench contains 3,654 multimodal jailbreak samples and is used to evaluate 11 recently released VLMs.Source: §1 · p. 2

2. Related Work

VLMs combine pretrained language models with visual encoders, bringing new vulnerabilities alongside their multimodal capabilities. Prior jailbreak work uses deceptive or adversarial images and manual or white-box approaches; IDEATOR instead generates diverse image-text pairs for black-box attacks.

2.1. Large Vision-Language Models (VLMs)

  • VLMs typically map visual features into an LLM’s token space through an alignment module, and visual integration introduces vulnerabilities.Source: §2.1 · p. 2

2.2. Jailbreak Attacks on VLMs

  • Prior attacks include deceptive text in images, harmful text rendered as images, query-relevant images, adversarial optimization, and dual-modality image-prefix and text-suffix methods.Source: §2.2 · p. 2
  • Existing methods rely on manual pipelines or white-box access, whereas IDEATOR is training-free and generates diverse image-text pairs for black-box attacks.Source: §2.2 · p. 2

3. Proposed Attack

IDEATOR models a black-box attacker VLM that generates multimodal jailbreak prompts and adapts them through victim interactions. Its design combines iterative refinement with parallel exploration, using open-source models in the main experiments for transparency and reproducibility.

3.1. Threat Model

  • The attacker has conversation history and probes the victim through black-box interactions, while the victim processes only the current turn.The attacker infers behavioral patterns and vulnerabilities without access to the victim’s internal architecture.Source: §3.1 · p. 3

3.2. IDEATOR

  • IDEATOR uses an attacker VLM to generate JSON responses containing analysis, image prompts, and text prompts for interactions with the victim.The image and text prompts are crafted to elicit harmful outputs while bypassing safety mechanisms.Source: §3.2 · p. 3

3.2.1. Formalization

  • In the first round, the attacker VLM processes the jailbreak goal without image input and produces structured JSON containing adversarial prompts.The image prompt is sent to a text-to-image model; the resulting image and text prompt are then input to the victim VLM.Source: §3.2.1 · p. 3

3.2.2. Prompt Design

  • The system prompt assigns the attacker a red-team role, constrains its output to JSON, and uses in-context exemplars to guide adversarial responses.Chain-of-Thought analysis of prior victim responses supports iterative strategy refinement; response post-processing and previous-round images reinforce interaction coherence.Source: §3.2.2 · pp. 3–4

3.2.3. Breadth-Depth Exploration

  • IDEATOR combines depth, which refines one strategy using victim feedback, with breadth, which launches diverse attacks to find a wider range of vulnerabilities.The authors state that this combination avoids over-reliance on one strategy and increases extensiveness and flexibility.Source: §3.2.3 · p. 4

3.2.4. Attacker Model Selection

  • The main experiments use MiniGPT-4 (Vicuna-13B) as the attacker and Stable Diffusion 3 Medium for image generation, prioritizing open-source models for transparency and reproducibility.The authors describe Vicuna as more permissive than LLaMA for crafting adversarial content and MiniGPT-4 as customizable through its system prompt and conversation template.Source: §3.2.4 · p. 4

4. IDEATOR Evaluation Experiments

IDEATOR was evaluated for attack effectiveness, transferability, and exploration and modality choices, with ASR judged through manual review of victim outputs. Experiments also visualized generated prompts and tested their performance across target VLMs.

4.1. Experimental Setup

  • Evaluation used MiniGPT-4, 100 randomly selected AdvBench goals, and VAJM’s 40 harmful instructions; manual ASR review required relevant, actionable harmful outputs.The main experiments used breadth Nb = 7 and depth Nd = 3.Source: §4.1 · pp. 4–5

4.2. Attack Effectiveness

  • 94% ASR: training-free, black-box IDEATOR matches white-box UMK and exceeds black-box MM-SafetyBench by 28% on 100 AdvBench samples.The highest ASR among unimodal white-box attacks was 85% for GCG-V; MM-SafetyBench recorded 66%.Source: Table 1 · p. 5, §4.2 · p. 5

4.3. Cross-Model Transferability

  • IDEATOR transfers at 82.0% ASR on LLaVA, 88.0% on InstructBLIP, and 75% on Chameleon, above MM-SafetyBench’s 46.0%, 29.0%, and 22.0%.The same image-text pairs generated on MiniGPT-4 were used against the three target models.Source: Table 2 · p. 5, §4.3 · p. 5

4.4. Visualization and Ablation

  • 45.0% to 94.0% ASR: increasing breadth and depth from Nb = 1, Nd = 1 to Nb = 7, Nd = 3 produced the strongest reported configuration.Figure 4 illustrates attack-image diversity, iterative refinement, and a successful image-text attack.Source: §4.4 · pp. 5–6, Table 3 · p. 6, Figure 4 · p. 6

Which Modality Is More Effective: Textual or Visual?

  • Combined text-image attacks achieved the highest ASR with the fewest queries; image-only attacks used fewer queries than text-only attacks but were generally less effective.Text attacks were more often rejected on crime-related topics, while image attacks were less effective for hate speech or self-harm.Source: Which Modality Is More Effective:… · p. 6

5. VLJailbreakBench

VLJailbreakBench evaluates multimodal jailbreak robustness with a two-tier dataset spanning safety topics and attack difficulty. Its challenge set shows high attack success rates across most evaluated VLMs, including commercial models.

5. VLJailbreakBench

  • VLJailbreakBench targets realistic multimodal adversarial scenarios, addressing benchmarks focused on harmful-content detection or text-based transfer attacks.The authors frame it as a practical evaluation of VLM vulnerabilities and safety limitations.Source: §5 · pp. 6–7

5.1. Benchmark Overview

  • The benchmark contains 3,654 jailbreak samples from 916 harmful queries, organized into base and challenge sets across 12 safety topics and 46 subcategories.Each query has one base-set pair and three challenge-set pairs; experts in the humanities and social sciences helped extend the safety taxonomy.Source: §5.1 · p. 7

5.2. Dataset Generation

  • The pipeline filters initial harmful queries, generates jailbreak pairs with IDEATOR, and retains successful instances using victim VLMs.Base-set attacks use MiniGPT-4 against LLaVA-1.5; challenge-set attacks use Gemini-1.5-Pro against GPT-4o-mini, with three successful samples retained per query.Source: §5.2 · pp. 7–8

5.3. Benchmarking Results

  • 72.21% is the challenge-set ASR on GPT-4o Mini, the highest among evaluated models; Claude-3.5-Sonnet has the lowest reported ASR at 19.65%.The challenge-set results cover 11 VLMs and 12 safety topics.Source: Table 4 · p. 8, §5.3 · p. 8

6. Conclusion

The paper presents IDEATOR as a black-box method for generating multimodal jailbreak pairs and reports strong attack success and transfer results. It also introduces a 3,654-sample benchmark that reveals safety gaps across 11 VLMs.

6. Conclusion

  • IDEATOR achieves a 94% success rate on MiniGPT-4 with 5.34 average queries, and transfer success rates of 82%, 88%, and 75% on LLaVA, InstructBLIP, and Chameleon.The method uses VLMs as red-team models to autonomously generate adversarial image-text pairs.Source: §6 · p. 8
  • VLJailbreakBench comprises 3,654 multimodal jailbreak samples for evaluating 11 state-of-the-art VLMs.The reported GPT-4o and Claude-3.5-Sonnet ASRs are 46.31% and 19.65%, respectively.Source: §6 · p. 8
  • IDEATOR is presented as a black-box jailbreak method for uncovering safety vulnerabilities in VLMs.The authors characterize its automated generation of adversarial image-text pairs as scalable.Source: §6 · p. 8

A. Additional Experimental Results

IDEATOR performs competitively on the VAJM evaluation set, with results varying across harmful-instruction categories. The paper also frames its attack space as expanding with exploration breadth and depth and encompassing multiple attack strategies.

A. Additional Experimental Results

B. Empirical Understanding

  • With infinite exploration breadth and depth, the defined attack set represents IDEATOR’s theoretical limit; increasing finite breadth and depth is expected to approach it.The paper states this as an ideal progression, not an empirical guarantee.Source: B. Empirical Understanding · p. 11
  • The theoretical attack set includes query-relevant typographic attacks and other strategies such as roleplay and emotional manipulation.The paper describes the typographic attacks as comparable to MM-SafetyBench strategies and attributes broader strategy diversity to the attacker VLM’s pretraining, modeling capabilities, and system prompts.Source: B. Empirical Understanding · p. 11

C. Additional Visualization Results

The visual examples show IDEATOR attacks across topics and successful jailbreaks against GPT-4o. Preliminary defense experiments report limited ASR reductions under AdaShield-S for the tested models.

C. Additional Visualization Results

  • Figure 11 uses rows for distinct attack topics, illustrating the diversity of IDEATOR’s attack strategies.The text also identifies three successful jailbreak attempts against GPT-4o in Figures 8–10.Source: C. Additional Visualization Results · p. 11
  • The GPT-4o examples include a simulated phishing email and detailed mail-fraud information, while another prompts potential actions after IoT-device access.The cited examples are presented as successful jailbreak attempts.Source: Figure 8 · p. 12, Figure 9 · p. 12, Figure 10 · p. 12

D. Evaluation against Defense Mechanisms

  • After AdaShield-S, IDEATOR’s ASR changes from 94.0% to 84.0% on MiniGPT-4, 82.0% to 73.0% on LLaVA, and 88.0% to 87.0% on InstructBLIP.The paper reports these as preliminary experiments and contrasts them with larger reductions for Figstep and MM-SafetyBench.Source: D. Evaluation against Defense Mechanisms · p. 12

E. VLJailbreakBench Data Statistics

VLJailbreakBench covers a broad range of safety risks across its base and challenge sets. Its challenge examples and IDEATOR images span subtle scenarios and varied attack topics.

E. VLJailbreakBench Data Statistics

  • IDEATOR attack images cover topics including computer viruses, identity theft, promoting anorexia, and committing fraud.Source: Figure 11 · p. 13
  • Table 6 reports ASR before and after applying AdaShield-S, with parenthetical values indicating absolute changes.Source: Table 6 · p. 13
  • Gemini-2.0-Flash has a 53.38% ASR, while Claude-3.5-Sonnet has a 1.09% ASR among commercial models.The passage also reports a 7.97% average ASR for source models.Source: E. VLJailbreakBench Data Statistics · p. 13
  • The dataset spans economic harm, ethical issues, hate speech, cyberattacks, privacy violations, and violent behavior.The categories and subcategories organize the base and challenge sets.Source: E. VLJailbreakBench Data Statistics · pp. 12–13
  • Challenge-set examples feature subtle, sophisticated attacks intended to test the limits of current defenses.Figure 12 showcases these examples as complex scenarios for testing VLM robustness.Source: E. VLJailbreakBench Data Statistics · p. 13

F. Benchmarking Results on the Base Set

The base set evaluates fundamental VLM vulnerabilities across safety topics. Qwen2-VL has the highest average ASR among open-source models reported in the passage.

F. Benchmarking Results on the Base Set

  • Qwen2-VL has a 35.04% average ASR, the highest vulnerability among open-source models on the base set.Source: F. Benchmarking Results on the Base Set · p. 13
  • Table 7 summarizes base-set and challenge-set samples by category and subcategory.Source: Table 7 · p. 14
  • Figure 12 shows complex challenge-set scenarios used to test VLM robustness.Source: Figure 12 · p. 15
  • Table 8 measures ASR for 11 VLMs across 12 safety topics and reports the average across topics.Safety topics and some model names are abbreviated in the table.Source: Table 8 · p. 15
Loading 2411.00827v6…