PaperlayerView exact revision on arXiv

Source-linked AI summary

Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?

Shuo Chen, Zhen Han, Bailan He, Zifeng Ding, Wenqian Yu, Philip Torr, Volker Tresp, Jindong Gu

arXiv:2404.03411v2cs.LGcs.CLcs.CR

TL;DR

Jailbreak attacks can bypass language-model safeguards, but universal evaluation and comprehensive testing of proprietary MLLMs have been lacking. The paper builds a benchmark and evaluates attacks across proprietary and open-source models. GPT-4 and GPT-4V are more robust than open-source models in the reported experiments, and visual jailbreak transferability is relatively limited.

  • Problem

    The absence of a universal benchmark and comprehensive evaluation of proprietary models, especially GPT-4V, leaves their robustness against existing jailbreak attacks unknown.

    Source: §1 · p. 1

  • Method

    The paper builds a 1,445-question benchmark spanning 11 safety policies and red-teams 11 LLMs and MLLMs using textual and visual attacks.

    Source: §1 · p. 2, §2.1 · p. 2

  • Results

    GPT-4 and GPT-4V are more robust than open-source models in most tested scenarios, and visual jailbreak methods show relatively limited transferability.

    Source: §3 · p. 4

  • Takeaways & Limitations

    The study finds that current visual jailbreak methods are harder to succeed against GPT-4V than textual methods are against GPT-4.

    Source: §4 · p. 5

Abstract

from arXiv · show

Various jailbreak attacks have been proposed to red-team Large Language Models (LLMs) and revealed the vulnerable safeguards of LLMs. Besides, some methods are not limited to the textual modality and extend the jailbreak attack to Multimodal Large Language Models (MLLMs) by perturbing the visual input. However, the absence of a universal evaluation benchmark complicates the performance reproduction and fair comparison. Besides, there is a lack of comprehensive evaluation of closed-source state-of-the-art (SOTA) models, especially MLLMs, such as GPT-4V. To address these issues, this work first builds a comprehensive jailbreak evaluation dataset with 1445 harmful questions covering 11 different safety policies. Based on this dataset, extensive red-teaming experiments are conducted on 11 different LLMs and MLLMs, including both SOTA proprietary models and open-source models. We then conduct a deep analysis of the evaluated results and find that (1) GPT4 and GPT-4V demonstrate better robustness against jailbreak attacks compared to open-source LLMs and MLLMs. (2) Llama2 and Qwen-VL-Chat are more robust compared to other open-source models. (3) The transferability of visual jailbreak methods is relatively limited compared to textual jailbreak methods. The dataset and code can be found https://github.com/chenxshuo/RedTeamingGPT4V

1 INTRODUCTION

Jailbreak attacks can bypass safety safeguards in text and visual inputs, while missing benchmarks and limited evaluations leave the robustness of proprietary MLLMs such as GPT-4V unclear. The paper builds a broad benchmark and red-teams proprietary and open-source models to compare their robustness and attack transferability.

1 INTRODUCTION

  • The study addresses the lack of a universal benchmark and comprehensive evaluation of proprietary models, especially MLLMs, by testing 11 models with 32 jailbreak methods.The methods include 29 textual and 3 visual attacks.Source: §1 · p. 1
  • GPT-4 and GPT-4V show greater robustness than open-source models, while visual jailbreak methods transfer less effectively than textual methods.Among open-source models, Llama2 and Qwen-VL-Chat are more robust; Llama2 can be more robust than GPT-4, and AutoDAN transfers better than GCG.Source: §1 · p. 2
  • The benchmark contains 1,445 harmful-behavior questions covering 11 safety policies for both LLMs and MLLMs.Source: §1 · p. 2
  • Red-teaming covers GPT-4, GPT-4V, and a range of state-of-the-art open-source models.Source: §1 · p. 2
  • The paper analyzes robustness of proprietary and open-source multimodal models against existing jailbreak methods.Source: §1 · p. 2

2 RED TEAMING GPT4 AGAINST JAILBREAK ATTACKS

The study evaluates textual and visual jailbreak attacks using surrogate models, a cross-model benchmark, and two complementary attack-success measures. Its methods include hand-crafted and optimized textual attacks alongside three visual attack approaches.

2.1 EXPERIMENTAL SETUP

  • The benchmark contains 1,445 collected harmful behaviors and questions spanning 11 usage policies, and the threat model tests transfer from open-source surrogate models to target models.Textual surrogates are Guanaco-7B, Llama2-7B, and Vicuna-7B; MiniGPT4-7B is the visual surrogate.Source: §2.1 · p. 2
  • The evaluation uses refusal-word detection and LLaMA-Guard judgments to measure attack success, reporting the LLaMA-Guard metric in the main paper.An attack counts as successful under refusal-word detection when predefined refusal words are absent; LLaMA-Guard checks whether generated content is harmful.Source: §2.1 · p. 2

2.2 RED TEAMING AGAINST TEXTUAL JAILBREAK

  • The textual evaluation includes 27 hand-crafted attacks—17 template-based and 10 function-based—and optimizes attack strings with GCG and AutoDAN.GCG trains a suffix; AutoDAN updates template tokens using genetic algorithms, with Guanaco-7B, Llama2-7B, and Vicuna-7B as surrogates.Source: §2.2 · p. 3

2.3 RED TEAMING AGAINST VISUAL JAILBREAK

  • The visual evaluation compares FigStep, VisualAdv, and ImageHijacks, using MiniGPT-4 as the surrogate for the two optimization-based methods.VisualAdv optimizes for harmful-content generation, while ImageHijacks optimizes for affirmative responses to harmful requests.Source: §2.3 · p. 3

3 DISCUSSION

GPT-4 and GPT-4V are generally more robust to jailbreaks than open-source models, though neither is invulnerable. Attack success varies by method and model, and visual jailbreak transfer is limited.

Which attack method is most powerful?

  • AutoDAN has higher success rates than GCG on open-source LLMs, especially Llama2-7B, while GPT-4 refuses almost all AutoDAN requests; FigStep outperforms VisualAdv and ImageHijacks across MLLMs.The authors suggest OpenAI safeguards may filter AutoDAN prompts and that token replacement may be insufficient to bypass them.Source: §3 · p. 4

How good is the current defense of the open-source model and closed-source model?

  • GPT-4 and GPT-4V are more robust than open-source models in most testing scenarios, but combined-model GCG still reaches a 2.39% success rate.FigStep reaches 35.99% on MiniGPT4-7B and 34.90% on Fuyu, while its success rate on GPT-4V is approximately 0.Source: §3 · p. 4

Does GPT-4 suffer more from visual jailbreak, compared to text modality?

  • GPT-4V does not show greater vulnerability to visual than textual jailbreaks in these experiments, and it refuses FigStep requests despite typography removing harmful context.The authors attribute this partly to input filtering and the fact that VisualAdv and ImageHijacks do not alter the harmful questions.Source: §3 · p. 4

How good is the transferability of jailbreak methods?

  • AutoDAN transfers better than GCG on open-source LLMs, while visual jailbreak methods have limited transferability and are ineffective against GPT-4V.For Fuyu, VisualAdv using MiniGPT4-7B as surrogate reaches 6.75%, below the 8.6% baseline; the authors say visual methods leave harmful questions intact.Source: §3 · p. 4

4 CONCLUSION

The study constructs a benchmark from existing jailbreak datasets and red-teams 11 LLMs and MLLMs. It finds GPT-4 and GPT-4V more robust than open-source models, while visual jailbreak methods are harder to succeed against GPT-4V.

4 CONCLUSION

  • The study constructs a benchmark from existing jailbreak datasets covering 11 usage policies, then evaluates 11 LLMs and MLLMs.The conclusion describes red-teaming proprietary and open-source models.Source: §4 · p. 4
  • Models are more robust compared to open-source models, and the gap is significant; compared to text modality, current visual jailbreak methods are hard to succeed on GPT-4V.Future work includes incorporating more jailbreak methods and datasets.Source: §4 · p. 5
  • The authors identify incorporating more jailbreak methods and datasets as future work.The stated scope is the study’s red-teaming of proprietary and open-source LLMs and MLLMs.Source: §4 · pp. 4–5

A RELATED WORK

Prior textual jailbreak work includes manually designed prompts and automatically optimized prompts. Visual jailbreak work manipulates images, but transfer to proprietary MLLMs such as GPT-4V remained unstudied.

Textual Jailbreak Attacks

  • Textual jailbreaks include hand-crafted prompts and automatic prompt optimization, using token-level gradients, genetic algorithms, or automated generation.Hand-crafted examples include low-resource languages, ciphers, in-context examples, and role-play prompts.Source: A RELATED WORK · p. 7

Visual Jailbreak Attacks

  • Earlier evaluations used different datasets and metrics, complicating fair comparison and reproduction; transfer attacks on proprietary MLLMs such as GPT-4V were also unstudied.Source: A RELATED WORK · p. 7
  • Visual jailbreak methods manipulate images through adversarial perturbations, embedded triggers, or typography paired with benign instructions.FigStep embeds harmful-question typography in images, while embedding-based attacks pair benign text with malicious image triggers.Source: A RELATED WORK · p. 7

B LLMS AND MLLMS USED IN THIS STUDY

The study uses 11 models: five LLMs and six MLLMs, including both closed-source and open-source systems. The listed models span different language, vision, and alignment designs.

B LLMS AND MLLMS USED IN THIS STUDY

C DATASET CONSTRUCTION

The benchmark compiles 1,445 harmful behaviors and questions from prior literature and covers 11 usage policies followed by Meta’s Purple LLaMA and OpenAI’s GPT-4.

C DATASET CONSTRUCTION

  • The dataset collects 1,445 harmful behaviors and questions from prior literature, including AdvBench, SafeBench, Qi et al., the GPT-4 technical report, and ToxicChat.The policies covered include Violence and Hate and Illegal Weapons.Source: C DATASET CONSTRUCTION · p. 8
  • Its questions cover 11 usage policies followed by Meta’s Purple LLaMA and OpenAI’s GPT-4.The policy coverage is shown in Table 4.Source: C DATASET CONSTRUCTION · p. 8

D EVALUATION METRICS

The study calculates attack success rate using refusal-word detection and an LLM judge, defining success per response and aggregating successful queries over all queries.

D EVALUATION METRICS

  • The study uses refusal-word detection and LLM judges to calculate attack success rate, among four evaluation categories discussed in prior work.The other categories are toxicity-detection APIs and human evaluation.Source: D EVALUATION METRICS · pp. 8–9
  • An attack succeeds under refusal-word detection when predefined refusal words are absent from the generated response.The study also uses LLaMA-Guard to judge whether generated content is harmful.Source: D EVALUATION METRICS · p. 9
  • Attack success rate is the ratio of successful jailbreak queries to total queries, with a function judging each response.The function returns 1 when a response is judged a successful jailbreak; LLaMA-Guard classifies responses as safe or unsafe and identifies violating policy subcategories.Source: D EVALUATION METRICS · p. 9

E ADDITIONAL EXPERIMENTAL RESULTS

The results show model and attack-specific variation: Llama2-7B is robust in baseline and GCG tests but vulnerable to some other attacks, while visual attacks also differ across models and methods.

E ADDITIONAL EXPERIMENTAL RESULTS

  • Llama2-7B is more robust than GPT-4 in the direct harmful-question baseline, while Guanaco-7B, Vicuna-7B, and LLaVA v1.5-7B show higher attack success rates.Table 5 reports success rates when harmful behaviors are given directly, without jailbreak methods.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 9, Table 5 · p. 10
  • Under GCG with Guanaco and Vicuna as surrogate models, attack success is 24.47% on ChatGLM2-6B and 2.39% on GPT-4; Llama2-7B has 0.14% harmful responses.With Guanaco-7B as surrogate, AutoDAN reaches 10.84% success on Llama2-7B, while GPT-4 shows better robustness; handcrafted attacks also expose vulnerabilities in both models.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 9, Table 6 · p. 10, Table 7 · p. 10
  • In visual transfer attacks, Fuyu fails to refuse 34.9% of harmful questions under FigStep; ImageHijacks reaches 52.35% on MiniGPT4-7B versus 35.99% for VisualAdv.The authors attribute the difference to optimization goals: VisualAdv maximizes harmful-content generation probability, whereas ImageHijacks maximizes affirmative-response probability.Source: E ADDITIONAL EXPERIMENTAL RESULTS · p. 9, Table 9 · p. 11, Table 10 · p. 11
  • Table 8 reports the attack success rate of 27 handcrafted textual jailbreak methods on both GPT-4 and open-source LLMs.Source: Table 8 · p. 10

Related papers

Loading 2404.03411v2…