PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts

Yuanwei Wu, Xiang Li, Yixin Liu, Pan Zhou, Lichao Sun

arXiv:2311.09127v2cs.CRcs.AIcs.LG

TL;DR

MLLM jailbreak research has focused primarily on adversarial model inputs, with less attention to API vulnerabilities. They extracted GPT-4V’s system prompt and used GPT-4V to generate jailbreak prompts; manual modification raised success to 99%, while appropriately tailored system prompts reduced jailbreak success. Human-refined attacks achieved a 98.7% success rate, while appropriately designed system prompts reduced jailbreak success rates.

  • Problem

    MLLM jailbreak research has focused primarily on adversarial model inputs, with less attention to API vulnerabilities.

    Source: §1 · p. 2

  • Method

    They extracted GPT-4V’s system prompt and used GPT-4V to generate jailbreak prompts; manual modification raised success to 99%, while appropriately tailored system prompts reduced jailbreak success.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Human-refined attacks achieved a 98.7% success rate, while appropriately designed system prompts reduced jailbreak success rates.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The findings identify potential security risks from system-prompt leakage and emphasize the need for robust protective measures.

    Source: §1 · p. 2

    The experiments were conducted before October 20, 2023, exclusively on the official OpenAI platform.

    Source: §4.4 · p. 6

Abstract

from arXiv · show

Existing work on jailbreak Multimodal Large Language Models (MLLMs) has focused primarily on adversarial examples in model inputs, with less attention to vulnerabilities, especially in model API. To fill the research gap, we carry out the following work: 1) We discover a system prompt leakage vulnerability in GPT-4V. Through carefully designed dialogue, we successfully extract the internal system prompts of GPT-4V. This finding indicates potential exploitable security risks in MLLMs; 2) Based on the acquired system prompts, we propose a novel MLLM jailbreaking attack method termed SASP (Self-Adversarial Attack via System Prompt). By employing GPT-4 as a red teaming tool against itself, we aim to search for potential jailbreak prompts leveraging stolen system prompts. Furthermore, in pursuit of better performance, we also add human modification based on GPT-4's analysis, which further improves the attack success rate to 98.7\%; 3) We evaluated the effect of modifying system prompts to defend against jailbreaking attacks. Results show that appropriately designed system prompts can significantly reduce jailbreak success rates. Overall, our work provides new insights into enhancing MLLM security, demonstrating the important role of system prompts in jailbreaking. This finding could be leveraged to greatly facilitate jailbreak success rates while also holding the potential for defending against jailbreaks.

1 Introduction

Prior MLLM jailbreak research focused mainly on adversarial inputs, with API vulnerabilities receiving limited attention. The paper reports extracting GPT-4V system prompts, using them to generate jailbreaks, and testing modified prompts as a defense.

1 Introduction

  • Safety fine-tuning aims to keep multimodal chatbots from producing inappropriate content in response to direct inquiries.The cited approaches include Ouyang et al. (2022), Korbak et al. (2023), and Bai et al. (2022a).Source: §1 · p. 1
  • Research on MLLM jailbreaks has emphasized adversarial examples in model inputs, while API vulnerabilities have received limited attention.The authors study black-box attacks through model APIs.Source: §1 · p. 2
  • The system prompt sets foundational response context and is confidential, unlike the user prompt that directs an immediate query.The paper distinguishes these roles in GPT-4V API interactions.Source: §1 · p. 2
  • SASP, which automates conversion of extracted system prompts into jailbreak prompts, achieved a 59% jailbreak success rate in GPT-4V; manual modification of its generated prompts raised success to 99%.Source: §1 · p. 2
  • The authors extracted GPT-4V system prompts through a carefully designed simulated, incomplete conversation, identifying an API vulnerability.They describe the vulnerability as a potential security risk in advanced AI systems.Source: §1 · p. 2
  • Appropriately tailored system prompts can substantially reduce the success rate of jailbreak attempts.The paper presents this as a potential defensive use of system prompts.Source: §1 · p. 2

2 Related Work

Prior adversarial work includes image perturbations for MLLMs and text-prompt attacks for LLMs. The paper draws on natural-language feedback methods to use GPT-4 and GPT-4V responses in generating jailbreak prompts from system-prompt vulnerabilities.

2.1 Text-based adversarial attacks

2.2 Prompting via Natural Language Feedback

  • Natural-language feedback has been used to refine model outputs, and this work uses GPT-4 feedback informed by GPT-4V responses and system prompts to generate jailbreak prompts.The authors say this lets GPT-4 identify effective prompts using insights from GPT-4V’s responses.Source: §2.2 · p. 3

3 Methodology

The methodology extracts GPT-4V’s system prompt with a text-image theft prompt, then uses it to generate and enhance jailbreak prompts. SASP has three stages: system prompt access, self-adversarial jailbreak, and jailbreak prompt enhancement.

3.1 System Prompt Theft

  • 72% attack success rate was achieved by a meta-theft prompt combining an incomplete simulated conversation, a completion request, and a random image trigger.The authors report that paraphrasing a system prompt for SASP can bypass the target model’s limits, suggesting verbatim recovery may not be necessary.Source: §3.1 · pp. 3–4

3.2 Self-Adversarial Iteration

  • Within two iterations, SASP achieves about 39% jailbreak success as GPT-4V iteratively revises prompts using target responses.The process continues until a jailbreak succeeds or the predetermined iteration limit is reached; the authors hypothesize that using the model’s own prompt helps expose its safety alignment.Source: §3.2 · p. 4

3.3 Jailbreak Prompt Enhancement

  • 99% ASR is reported after combining prefix injection, refusal suppression, hypothetical scenarios, and emotional appeals.The authors say hypothetical scenarios can divert attention from system instructions, while emotional appeals exploit morality learned during pretraining.Source: §3.3 · pp. 4–5

4 System Prompt Facilitated Jailbreak

In facial-recognition tests, GPT-4V resisted baseline prompts, while SASP-derived prompts substantially increased attack rates. Further experiments examined iteration, dependence on system-prompt access, and modified prompts as a defense.

4.1 Experiment Setup

  • The facial-recognition study used 100 images and tested English, Chinese, and Thai jailbreak prompts.The authors selected images from a dataset and hypothesized that the depicted public figures were likely included in GPT-4V’s training data.Source: §4.1 · p. 5
  • ASR counts any real human name as an attack success, RSR measures correct identification, and DSR counts explicit refusals; DSR is not simply ASR’s inverse.Outputs that are neither a real name nor a direct refusal count as unsuccessful for both attack and defense.Source: §4.1 · pp. 5–6

4.2 Implementation Details

  • GPT-4V used the OpenAI platform’s default settings; LLaVA-1.5v used temperature 0.2, 512 maximum new tokens, and padded image aspect ratio.Baseline methods used their default hyperparameters.Source: §4.2 · p. 6

4.3 Main Results

  • Baseline Attack produced 0% ASR and RSR and 100% DSR across all tested languages, indicating GPT-4V defended against prefix injection and refusal suppression.Source: §4.3 · p. 6
  • 99% ASR and 95% RSR resulted from English SASP prompts augmented with Prefix Injection, Refusal Suppression, and Hypothetical Scenario; unmodified SASP reached up to 59% English ASR.SASP did not perform well in Chinese and Thai.Source: §4.3 · p. 6, Table 1 · p. 6

4.4 Analytical Experiment

  • Within a maximum of two iterations, jailbreak-prompt generation rose from 12% after one iteration to 27% after two, while generated prompts averaged 63% ASR.The authors report that feedback provides insights for improving prompt generation.Source: §4.4 · p. 6
  • Table 2 compares direct-input and system-prompt-recall rates for default versus safety system prompts; the experiments were conducted before October 20, 2023.Bold values identify the lowest ASR or RSR, or the highest DSR.Source: §4.4 · p. 6, Table 2 · p. 7
  • Without access to the target model’s system prompt, SASP generated no effective jailbreak prompts within two iterations.Source: §4.4 · p. 7

5 System Prompt Enhanced Defence

Section 5 evaluates manually designed system prompts for safer MLLM outputs and introduces a system-prompt recall method. On LLaVA-1.5v, recall generally produces more benign outputs when the model appears to overlook a safety prompt.

5 System Prompt Enhanced Defence

  • The authors manually designed system prompts for safe outputs and tested their efficacy.The section also introduces a system-prompt recall method intended to produce harmless content consistently.Source: §5 · p. 7

5.1 Experiment Setting

  • The study selected 170 pictures and used LLaVA-1.5v to evaluate human recognition and sensitive inference, with human evaluation of results.The system prompts are provided in the Appendix.Source: §5.1 · p. 7

5.2 Evaluation Metrics

  • A sensitive-inference attack counts as successful when the model accurately infers any of six sensitive traits from a real person’s image.The traits are religious attributes, education level, political attributes, financial situation, personality types, and mental state.Source: §5.2 · p. 7

5.3 Recall the System Prompt

  • Safe-output prompts did not reduce harmful outputs from LLaVA-1.5v-7b at 8-bit quantization.The outputs were as potentially harmful as those without the system prompt.Source: §5.3 · p. 7
  • Querying content policy before a potentially inappropriate request generally yields more benign outputs, especially when safety prompts appear overlooked.The recall method does not notably improve outputs that are already harmless.Source: §5.3 · p. 7
  • Table 3 contrasts direct-input and system-prompt-recall rates for models with safety prompts or default prompts, using ASR, RSR, and DSR columns.Source: Table 3 · p. 8

6 Conclusion

The conclusion presents system-prompt leakage in GPT-4V as a vulnerability that enabled SASP, a jailbreak method derived from compromised prompts. Human refinement raised attack success to 98.7%, while well-designed prompts could also reduce jailbreak success.

6 Conclusion

  • 98.7% attack success was achieved by refining GPT-4V-derived jailbreak prompts with human intervention.The prompts were derived from compromised system prompts using GPT-4 as a red-teaming tool against itself.Source: §6 · p. 8
  • Well-designed system prompts can greatly reduce the success of jailbreak attacks.Source: §6 · p. 8
  • The findings highlight system prompts’ roles in both facilitating and thwarting MLLM jailbreak attempts.Source: §6 · p. 8
Loading 2311.09127v2…