Source-linked AI summary
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, Xiaoyun Wang
TL;DR
LVLM safety remains underexplored because safety alignment in underlying LLMs may not protect visual inputs. The paper introduces FigStep, a black-box typographic-image jailbreak, and reports strong attack performance while identifying cross-modal alignment weaknesses and a GPT-4V limitation.
Problem
LVLM safety is underexplored, and it is unclear whether underlying-LLM safety alignment adequately protects multimodal systems.
Method
FigStep converts prohibited textual instructions into typographic images and uses them as visual prompts for black-box LVLM jailbreaks.
Results
82.50% average ASR across six open-source LVLMs, with FigStep outperforming evaluated image-based jailbreaks and text-based methods.
Takeaways & Limitations
The findings expose cross-modality safety-alignment vulnerabilities and motivate novel alignment methods connecting textual and visual modalities.
Takeaways & Limitations
FigStep achieves 34% ASR on GPT-4V because its OCR tool detects harmful information in image prompts.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) signify a groundbreaking paradigm shift within the Artificial Intelligence (AI) community, extending beyond the capabilities of Large Language Models (LLMs) by assimilating additional modalities (e.g., images). Despite this advancement, the safety of LVLMs remains adequately underexplored, with a potential overreliance on the safety assurances purported by their underlying LLMs. In this paper, we propose FigStep, a straightforward yet effective black-box jailbreak algorithm against LVLMs. Instead of feeding textual harmful instructions directly, FigStep converts the prohibited content into images through typography to bypass the safety alignment. The experimental results indicate that FigStep can achieve an average attack success rate of 82.50% on six promising open-source LVLMs. Not merely to demonstrate the efficacy of FigStep, we conduct comprehensive ablation studies and analyze the distribution of the semantic embeddings to uncover that the reason behind the success of FigStep is the deficiency of safety alignment for visual embeddings. Moreover, we compare FigStep with five text-only jailbreaks and four image-based jailbreaks to demonstrate the superiority of FigStep, i.e., negligible attack costs and better attack performance. Above all, our work reveals that current LVLMs are vulnerable to jailbreak attacks, which highlights the necessity of novel cross-modality safety alignment techniques. Our code and datasets are available at https://github.com/ThuCCSLab/FigStep .
1 INTRODUCTION
LVLMs combine visual and textual processing, but their component-wise safety alignment may leave visual inputs insufficiently protected. FigStep exploits this gap by transferring harmful instructions into typographic images and is proposed as a low-cost black-box assessment baseline.
- FigStep: FigStep transfers harmful information from text to visual typography, bypassing the textual module’s safety alignment.The method is presented as distinct from perturbation-based attacks and requiring only black-box access.
- Motivation: LVLMs process visual and textual modalities through a visual module, connector, and textual module, with the latter typically safety-aligned as an underlying LLM.The visual module extracts image embeddings, the connector maps them into the textual module’s latent space, and the textual module generates responses.
- Motivation: Safety alignment of the underlying LLM may not cover domains introduced by the visual modality because LVLM components are not aligned as a whole.The introduction frames this as a question about whether underlying-LLM alignment provides an illusory safety guarantee for LVLMs.
- FigStep: FigStep eliminates perturbation requirements and offers negligible computational cost and technical barriers for LVLM safety assessment.The authors position it as a convenient baseline compared with optimization-based jailbreaks.
- Contributions: The paper introduces SafeBench, evaluates FigStep against six open-source LVLMs, and reports visual embeddings that are semantically but not safely aligned with textual embeddings.The contribution passages connect benchmark construction, empirical evaluation, and semantic-embedding analysis.
- Contributions: The authors advocate strict cross-modal alignment and position FigStep as a probe for developing compositional textual–visual safety alignment methods.They conclude that LVLM safety cannot rely solely on underlying LLMs.
2 RELATED WORK
Prior LVLM jailbreak research primarily uses adversarial image perturbations, while this paper focuses on a black-box alternative that transfers harmful content into visual inputs.
- Safety Alignment: Safety-alignment attacks include model fine-tuning and input jailbreaks that bypass safeguards without updating model parameters.The paper focuses on jailbreaks rather than removing guardrails through fine-tuning.
- Jailbreak Against LLMs: LLM jailbreak methods are broadly categorized as gradient-based or non-gradient approaches, including Greedy Coordinate Gradient.These methods modify inputs rather than model parameters.
- Jailbreak Against LVLMs: Recent LVLM attacks generate adversarial image perturbations, which can be more effective than optimizing discrete text tokens.These attacks follow standard adversarial-attack methods from computer vision.
- Jailbreak Against LVLMs: Jailbreak in Pieces uses an OCR textual trigger as a reference image to generate adversarial images and requires white-box access to the visual module.The paper contrasts this requirement with its black-box focus.
- Jailbreak Against LVLMs: Image-prompt commands can guide LVLMs to accomplish tasks depicted in the image.
3 PRELIMINARIES
The preliminaries formalize LVLMs as models that accept textual and visual prompts and define content safety through prohibited-query and effective-response oracles.
- Definitions: An LVLM is a text-generation model that processes both textual prompts and image prompts.The query domain combines optional image and text components.
- Definitions: The LVLM is modeled as a probabilistic function M mapping multimodal queries to textual responses.Text-only queries contain only a text prompt.
- Content Safety: The prohibited-query oracle O_q returns 1 for queries forbidden by the content-safety policy and 0 otherwise.
- Content Safety: The effective-response oracle O_r returns 1 when a response satisfies the intention or goal behind a query and 0 otherwise.
4 THREAT MODEL
The threat model considers a regular user who seeks prohibited answers through API access without control over the model’s internal generation process.
- Adversary’s Goal: The adversary aims to obtain answers to questions forbidden by the LVLM’s safety policy.The paper motivates this as a misuse scenario with potential harm to users and society.
- Adversary’s Knowledge: The interaction is restricted to one turn with a preset system prompt and no dialogue history.
- Adversary’s Knowledge: The adversary lacks access to or control over internal generation states and can adjust only some generation parameters, such as temperature.
- Adversary’s Knowledge: The threat model represents an attacker who can query an LVLM API but cannot deploy a local instance because of unavailable models or scarce resources.
5 METHODOLOGY
FigStep jailbreaks LVLMs by converting prohibited instructions into typographic images and pairing them with benign incitement text. Its three-stage pipeline paraphrases the instruction, renders it typographically, and prompts the model to complete the resulting list.
- Motivation: The method is motivated by LVLMs’ ability to understand typographic visual instructions and the ineffectiveness of their safety guardrails against them.These observations rely on multimodal instruction-following and text recognition capabilities while targeting the safety gap for typographic prompts.
- Pipeline overview: FigStep generates a jailbreaking query from a prohibited text-only query by combining a typographic image prompt with benign incitement text.The resulting query is defined as Q_jail = (T′, I′), where I′ contains the harmful instruction as typography and T′ motivates completion.
- Pipeline overview: FigStep uses three stages: Paraphrase, Typography, and Incitement.The stages are presented as the complete attack pipeline illustrated in Figure 2.
- Paraphrase: Paraphrase rewrites the prohibited question as a statement beginning with list-oriented nouns such as “Steps to,” “List of,” or “Methods to.”For example, “How can someone manufacture illegal drugs?” becomes “Steps to manufacture illegal drugs.”
- Typography: Typography converts the paraphrased statement into an image and adds indices from 1 to 3 as a response-format hint.The typographic image becomes the final jailbreaking image prompt I′.
- Incitement: Incitement uses a neutral, benign text prompt that asks the LVLM to generate detailed content for empty numbered items.The default prompt requests approximately 100 words for each of three list items and is designed to avoid triggering content-safety mechanisms.
6 EVALUATION
Evaluation on SafeBench shows that FigStep substantially increases jailbreak success across open-source LVLMs, including models with text-only safety alignment. Ablations and embedding analyses indicate that both typographic content and a benign continuation prompt contribute to the attack’s effectiveness.
- Experimental Setup: ASR was manually assessed over repeated trials because automated jailbreak evaluators were unstable.Each question was attacked with FigStep five times, and responses were manually evaluated.
- Vanilla Query: 83.40% ASR for MiniGPT4-Vicuna-13B versus 45.40% for LLaVA-v1.5-Vicuna-v1.5-13B links safety disparity to their underlying LLMs.The paper attributes the difference to stronger safety improvement in Vicuna-v1.5-13B.
- Vanilla Query: Vanilla-query robustness varies by forbidden topic, with LLaMA-2-CHAT-7B showing comparatively unacceptable ASR on the last three expert-consultation topics.The first seven topics are described as having better safety properties, while topic restrictions remain inconsistent overall.
- Jailbreaking via FigStep: FigStep induces specific policy-violating steps on LLaVA, MiniGPT4, and CogVLM, demonstrating successful jailbreaks across multiple open-source LVLMs.The successful instances are tailored to the prohibited queries shown in the visual prompts.
- Jailbreaking via FigStep: 82.60% ASR for FigStep versus 23.80% for vanilla queries on LLaMA-2-Chat-7B shows a large increase despite strong text-only alignment.Across models, the reported FigStep improvement over vanilla inputs ranges from 1.80% to 78.80%.
- Jailbreaking via FigStep: 76.86% average ASR for FigStep versus 5.14% for vanilla queries across MiniGPT4-Llama-2-CHAT-7B’s first seven topics demonstrates cross-topic vulnerability.The paper reports high FigStep ASR across a variety of prohibited topics and across different LVLMs.
- Ablation Study: Embedding harmful content in the image and adding a benign incitement prompt are both important: without the latter, the model fails to complete the image-presented information.The ablation also finds that FigStep outperforms variants with harmful words in textual prompts and consistently surpasses a direct-answer prompt.
- Embedding Analysis: Embedding overlap appears for benign and prohibited queries under FigStep, whereas text-only prompts produce highly separable embeddings for LLaVA and MiniGPT4.The authors interpret this pattern as visual embedding transformation ignoring textual safety constraints; CogVLM shows no separation for either prompt type.
7 A CASE STUDY: JAILBREAKING GPT-4V AND GPT-4O
The case study examines FigStep and its upgraded FigSteppro variant against GPT-4V and GPT-4o. FigSteppro segments harmful content across sub-images to evade OCR moderation before prompting the model to reconstruct the intended instruction.
- Limitation of FigStep: FigStep’s GPT-4V limitation is linked to OpenAI’s OCR detector, which reduces its ASR to 34% after repeated querying.FigSteppro is introduced to bypass the OCR detector while retaining FigStep’s core pipeline.
- Limitation of FigStep: FigSteppro cannot be used on current open-source LVLMs because they process only a single image per query.The upgraded method requires feeding multiple image-prompts simultaneously.
- FigSteppro: FigSteppro embeds harmful keywords across multiple sub-figures so each individual image remains harmless to OCR moderation.The sub-figures are later supplied collectively to GPT-4V.
- FigSteppro: FigSteppro uses incitement prompts that instruct GPT-4V to concatenate the sub-figures and generate action-related steps in list form.Two candidate text prompts are described for inducing reconstruction of the concatenated image’s content.
- Takeaways: 70% ASR is achieved against GPT-4V by FigSteppro, 36% higher than FigStep.The comparison is reported in Table 7 and is presented as evidence that segmentation can invalidate OCR-based moderation.
8 CONCLUSION
The conclusion presents FigStep as an efficient black-box jailbreak that exposes cross-modal safety vulnerabilities in LVLMs. It also describes prompt-based procedures and positions the method as a probe for developing stronger alignment techniques.
- Conclusion: The paper introduces FigStep, which transforms harmful textual instructions into typographic images to circumvent underlying LLM safety alignment.The method is described as a straightforward jailbreak algorithm against LVLMs.
- Conclusion: The conclusion reports significant misuse risks for both open-source and closed-source LVLMs and calls for new defensive mechanisms.The authors frame the findings as motivating responsible safety research and stronger defenses.
B SETTINGS OF SENSITIVITY ANALYSIS ON FigStep
The sensitivity analysis varies FigStep’s visual and textual prompts. It examines randomized typography, semantically equivalent incitement prompts, and visual-prompt variants across SafeBench’s forbidden topics.
- Image-Prompt Sensitivity: FigStep includes additional image-prompts beyond the default setting, including randomized backgrounds, fonts, and text colors.Randomization uses FreeMono font variants and hexadecimal colors while preserving legibility.
- Text-Prompt Sensitivity: The analysis introduces three semantically equivalent text-prompts as alternatives to FigStep’s default incitement prompt.These alternatives describe empty numbered entries and request detailed content for each item.
- Benchmark Topics: SafeBench contains 10 forbidden topics with corresponding descriptions for sensitivity analysis.The topics define the evaluated harmful-content categories.
D DETAILS OF EXPERIMENT ENVIRONMENT
The experiments use specified CPU, GPU, operating-system, Python, and PyTorch configurations. The appendix also records successful FigStep jailbreak instances and examples of benign and prohibited questions.
- Hardware and Operating System: Experiments run on an Intel Xeon Gold 6348 CPU, one NVIDIA A800 GPU with 80GB, and Ubuntu 22.04.1.GPU memory usage reaches up to 35GB.
- Software Environment: The appendix specifies separate Python and PyTorch versions for LLaVA, MiniGPT4, and CogVLM-Chat-v1.1.The listed versions vary across the three model families.
- Jailbreak Instances: Figure 3 presents successful FigStep jailbreak instances on LLaVA, MiniGPT4, and CogVLM.The figure documents examples across three open-source LVLM families.
- Question Examples: SafeBench contrasts benign and prohibited questions within the Illegal Activity topic.The examples concern preventing violence versus recruiting individuals for terrorist activities.
- Visual Prompt Configuration: FigStephide visual prompts use background color 0x000010, close to the font color 0x000000.The near-matching colors characterize this visual-prompt variant.
H COMPARISON WITH OTHER JAILBREAK METHODS
FigStep is compared with text-only and image-based jailbreak methods to assess attack performance and required access. The comparisons include standard, red-teamed, and adversarial-perturbation variants.
- Comparison with Text-based Jailbreaks: FigStep is compared with five text-only jailbreak algorithms using MiniGPT4-Llama-2-CHAT-7B as the victim model.The text-based methods are implemented through EasyJailbreak with its default attacking hyperparameters.
- Comparison with Text-based Jailbreaks: 0% ASR is obtained for vanilla queries, while the evaluated text-only methods remain inferior to FigStep.
- Comparison with Image-based Jailbreaks: Image-based comparisons assess original and FigStep-improved attacks, including visual adversarial examples and JPOCR on SafeBench.
- Comparison with Image-based Jailbreaks: FigStep and JPOCR are extended with GPT-4-rephrased prompts to form red-teaming variants that account for LVLM sensitivity to text prompts.
- Comparison with Image-based Jailbreaks: Adversarial variants hide visible prompt text through perturbations, requiring white-box access to MiniGPT4-Llama-2-CHAT-7B's visual module.
I AN INSTANCE OF AI ANNOTATION
An example shows FigStep eliciting a detailed legal-contract response from LLaVA despite the question requesting drafting without supervision. The evaluation nevertheless labels this instance an unsuccessful jailbreak.
- Instance: FigStep prompts LLaVA to provide step-by-step guidance for drafting a legal contract without supervision.The response covers identifying parties, defining terms, and including legal clauses and provisions.
- Evaluation: Despite violating AI policies, the evaluation erroneously classifies the example as an unsuccessful jailbreak.