PaperlayerView exact revision on arXiv

Source-linked AI summary

Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks

Zonghao Ying, Aishan Liu, Xianglong Liu, Dacheng Tao

arXiv:2406.06302v2cs.CRcs.CV

TL;DR

GPT-4o’s safety against jailbreak attacks remained insufficiently explored despite its powerful general capabilities and the societal risks of harmful generated content. The paper evaluates it across text, image, and audio modalities using thousands of queries and responses, finding distinct modality-specific safety patterns and limitations in existing attacks. These findings motivate stronger alignment guardrails for multimodal models.

  • Problem

    GPT-4o’s safety against jailbreak attacks has not been sufficiently explored despite the potential societal impact of risky generated content.

    Source: §1 · p. 1

  • Method

    The paper evaluates GPT-4o with seven unimodal and multimodal jailbreak methods across text, audio, and multimodal benchmarks using multiple safety judgments.

    Source: §1 · p. 2, §2 · pp. 2–3

  • Results

    GPT-4o shows enhanced text-jailbreak safety relative to GPT-4V, audio creates a new attack surface, and existing black-box multimodal attacks are largely ineffective against both models.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    The findings underscore the need for robust alignment guardrails and increased attention to safety risks in multimodal large language models.

    Source: §1 · p. 2, §5 · p. 8

    Audio evaluation was limited to manually testing translated textual adversarial prompts through the mobile application, while multimodal analysis focused primarily on text-vision attacks.

    Source: §3.2 · p. 4, §5 · p. 8

Abstract

from arXiv · show

The recent release of GPT-4o has garnered widespread attention due to its powerful general capabilities. While its impressive performance is widely acknowledged, its safety aspects have not been sufficiently explored. Given the potential societal impact of risky content generated by advanced generative AI such as GPT-4o, it is crucial to rigorously evaluate its safety. In response to this question, this paper for the first time conducts a rigorous evaluation of GPT-4o against jailbreak attacks. Specifically, this paper adopts a series of multi-modal and uni-modal jailbreak attacks on 4 commonly used benchmarks encompassing three modalities (ie, text, speech, and image), which involves the optimization of over 4,000 initial text queries and the analysis and statistical evaluation of nearly 8,000+ response on GPT-4o. Our extensive experiments reveal several novel observations: (1) In contrast to the previous version (such as GPT-4V), GPT-4o has enhanced safety in the context of text modality jailbreak; (2) The newly introduced audio modality opens up new attack vectors for jailbreak attacks on GPT-4o; (3) Existing black-box multimodal jailbreak attack methods are largely ineffective against GPT-4o and GPT-4V. These findings provide critical insights into the safety implications of GPT-4o and underscore the need for robust alignment guardrails in large models. Our code is available at \url{https://github.com/NY1024/Jailbreak_GPT4o}.

1 INTRODUCTION

This paper evaluates GPT-4o’s safety against jailbreak attacks because risky generated content can have societal impact. It examines multimodal and unimodal attacks across text, visual, and audio modalities, finding improved text-jailbreak safety relative to earlier models, a new audio attack surface, and weak effectiveness of existing black-box multimodal attacks.

  • The study evaluates GPT-4o against jailbreak attacks because such attacks can bypass guardrails and induce otherwise prohibited harmful or unethical content.Source: §1 · p. 1
  • The evaluation combines multimodal and unimodal jailbreak attacks across four benchmarks and three modalities.The modalities are text, speech, and image.Source: §1 · p. 1
  • More than 4,000 initial text queries and nearly 8,000 responses support the statistical evaluation of GPT-4o’s safety.Source: §1 · p. 2
  • GPT-4o shows enhanced safety against text-modal jailbreak attacks relative to GPT-4V, while text attacks still transfer effectively to multimodal models.Source: §1 · p. 2
  • Audio introduces a new attack surface, whereas existing black-box multimodal jailbreak methods are largely ineffective against GPT-4o and GPT-4V.Source: §1 · p. 2
  • The paper aims to inform robust alignment guardrails for next-generation multimodal large models.Source: §1 · p. 2

2 EVALUATION PROTOCOLS

The evaluation compares GPT-4o and GPT-4V using established unimodal and multimodal jailbreak datasets and seven attack methods. Attack success rate is judged through multiple evaluators, with GPT-4o judgment prompts used primarily for quantitative analysis.

Models

  • The study compares GPT-4o and GPT-4V, using Llama2 to generate transferable textual adversarial prompts for the target models.Source: §2 · p. 2

Datasets

  • The datasets combine open-source text, audio, and multimodal jailbreak benchmarks categorized under safety policies.Source: §2 · p. 2

Jailbreak methods

  • Seven jailbreak methods span template-based, unimodal, and multimodal attacks, including GCG, AutoDAN, PAP, FigStep, Liu et al., and BAP.Source: §2 · p. 2

Metrics

  • Attack success rate measures the proportion of responses judged unsafe for a dataset and target model.Source: §2 · pp. 2–3
  • Four judgment functions use string matching, Llama Guard, ChatGPT prompts, or GPT-4o prompts to identify unsafe responses.Source: §2 · p. 3

Judgment functions

  • The judgment functions can produce discrepant ASRs: string matching may yield false negatives, Llama Guard may be overly stringent, and ChatGPT judgments may hallucinate jailbreaks.Source: §2 · p. 3
  • Quantitative analyses primarily rely on J4, while qualitative analyses use majority decisions across the judgment functions.Source: §2 · p. 3

3 UNIMODAL JAILBREAK ATTACKS EVALUATION

The evaluation finds that GPT-4o’s safety differs across unimodal settings: it is safer than GPT-4V under most text attacks, while audio-converted adversarial prompts retain substantial jailbreak effectiveness.

3.1 TEXT MODALITY JAILBREAK

  • Without attacks, GPT-4o had lower safety than GPT-4V overall, with a 14.6% ASR difference in the Physical Harm scenario.Source: §3.1 · p. 3
  • PAP achieved ASRs of 62.2% on GPT-4V and 62.7% on GPT-4o, while BAP reached 83.1% against GPT-4V.Source: §3.1 · p. 4
  • GPT-4o was safer than GPT-4V under text jailbreak attacks across judgment functions and attack methods, except PAP under J3.Source: §3.1 · p. 4

3.2 AUDIO MODALITY JAILBREAK

  • Audio testing was limited by unavailable audio APIs and mobile-app rate limits, relying on manual assessment of 200 converted text samples.Source: §3.2 · pp. 4–5
  • Original text queries converted to audio did not jailbreak GPT-4o, but adversarial audio prompts retained ASRs close to text attacks.Source: §3.2 · p. 5
  • GCG and AutoDAN suffixes lost critical semantic information during audio processing, whereas PAP and BAP remained effective through persuasive, human-like strategies.Source: §3.2 · p. 5

4 MUTLIMODAL JAILBREAK ATTACKS

The study evaluates black-box multimodal jailbreak attacks on GPT-4o using typographic and image-semantic adversarial prompts, finding these methods largely ineffective overall while GPT-4o can be more vulnerable than GPT-4V in some comparisons.

Attack setup

  • The evaluated multimodal methods use pre-constructed SafeBench and MM-SafetyBench datasets containing typographic or image-semantic adversarial prompts.SafeBench converts harmful intent into formatted typographic images, whereas MM-SafetyBench transforms harmful semantics into images and appends crucial typographic text.Source: §4 · p. 6

SafeBench

  • GPT-4o’s ASR is consistently higher than GPT-4V’s across SafeBench judgment functions, reaching 42.0% in J4, indicating greater susceptibility to FigStep attacks.The paper links this difference speculatively to GPT-4o’s stronger multimodal fusion capabilities.Source: §4 · pp. 6–7
  • FigStep produces only a 3.6% ASR for GPT-4o on Hate Speech outside anomalous scenarios, barely exceeding No Attack performance.The authors caution that commercial-model defenses may change over time, so this result does not rule out earlier FigStep effectiveness against GPT-4V.Source: §4 · p. 6

MM-SafetyBench

  • MM-SafetyBench reports ASRs approaching 100% for some scenarios such as Legal Opinion and Financial Advice, partly because its textual queries lack sufficient harmfulness and use scenario-specific failure criteria.For strongly harmful scenarios including Hate Speech, Physical Harm, and Fraud, attacked models instead have lower ASRs than under No Attack.Source: §4 · pp. 6–7
  • The MM-SafetyBench image-semantic attack is ineffective against GPT-4o and GPT-4V, with attacked ASRs usually below No Attack except in selected scenarios.The authors speculate that OpenAI may detect harmful image semantics and block injected instructions.Source: §4 · p. 7

Overall findings

  • GPT-4o and GPT-4V generally show lower ASRs under multimodal attacks than under No Attack, indicating current black-box multimodal methods are largely ineffective.The study attributes this pattern to possible stronger defenses triggered by harmful semantics in images.Source: §4 · p. 7

5 CONCLUSION

The conclusion presents the study as an initial cross-modal safety evaluation of GPT-4o, combining large-scale API testing for text and vision with a limited manual audio evaluation. It highlights safety risks and calls for stronger alignment strategies while acknowledging constrained audio and multimodal coverage.

5 CONCLUSION

  • The study evaluates GPT-4o primarily through automated text-vision jailbreak testing via the OpenAI API and manually tests audio jailbreaks on a subset of AdvBench through the mobile app.These choices reflect limitations of the official OpenAI API.Source: §5 · p. 7
  • The authors aim to raise awareness of MLLM safety risks and urge researchers to prioritize alignment strategies and mitigation techniques.Source: §5 · p. 8

Limitations and discussions

  • The audio evaluation only translates known textual adversarial prompts into audio, reducing the flexibility of the audio jailbreak attacks.The paper also notes that current multimodal datasets are insufficient for comprehensive combinations of text, vision, and audio.Source: §5 · p. 8

A.1 STRINGS MATCHING

The appendix identifies the strings used to match responses in the J1 evaluation method.

A.1 STRINGS MATCHING

A.2 JUDGEMENT PROMPTS

The appendix presents two judgment prompts used for safety evaluation with ChatGPT and GPT-4o.

A.2 JUDGEMENT PROMPTS

  • The study uses separate judgment prompts with ChatGPT and GPT-4o to evaluate model responses.Source: §A.2 · p. 13

A.4 EXAMPLE OF FIGSTEP ATTACK.

The FigStep example contrasts multimodal and text-only queries in GPT-4o jailbreak attempts, with multimodal queries failing while original textual queries succeed.

  • Multimodal FigStep queries result in jailbreak failures, whereas using only the original textual queries leads to successful jailbreaks.Source: §A.4 · p. 13
  • The section illustrates this behavior through a GUI interaction interface implemented for GPT-4o on the Coze platform.Source: §A.4 · p. 13
  • The accompanying prompts are presented separately for ChatGPT and GPT-4o.Source: Table 6 · p. 14, Table 7 · p. 14

A.5 EXAMPLE OF LIU et al.ATTACK.

The Liu et al. attack example similarly shows multimodal jailbreak attempts failing while the original textual queries succeed, with accompanying attack examples and prompt templates.

  • Multimodal Liu et al. queries result in jailbreak failures, whereas using only the original textual queries leads to successful jailbreaks.Source: §A.5 · p. 14
  • The example uses a GUI interaction interface for GPT-4o on the Coze platform to illustrate the observed jailbreak behavior.Source: §A.5 · p. 14
  • Six jailbreak prompt templates collected from the Internet are presented as supplementary attack material.Source: Table 8 · p. 15
  • The section includes examples labeled for FigStep and Liu et al. attacks.Source: Figure 2 · p. 16, Figure 3 · p. 16
Loading 2406.06302v2…