PaperlayerView exact revision on arXiv

Source-linked AI summary

Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey

Xuannan Liu, Xing Cui, Peipei Li, Zekun Li, Huaibo Huang, Shuhan Xia, Miaoxuan Zhang, Yueying Zou, Ran He

arXiv:2411.09259v2cs.CVcs.CL

TL;DR

Multimodal models can be jailbroken through vulnerabilities spanning different data types, raising security and reliability concerns. This survey organizes attacks and defenses across four lifecycle levels and reviews methods, evaluations, and model configurations. It synthesizes current coverage and identifies research challenges, including underexplored video and audio vulnerabilities and limited defense evaluation coverage.

Abstract

from arXiv · show

The rapid evolution of multimodal foundation models has led to significant advancements in cross-modal understanding and generation across diverse modalities, including text, images, audio, and video. However, these models remain susceptible to jailbreak attacks, which can bypass built-in safety mechanisms and induce the production of potentially harmful content. Consequently, understanding the methods of jailbreak attacks and existing defense mechanisms is essential to ensure the safe deployment of multimodal generative models in real-world scenarios, particularly in security-sensitive applications. To provide comprehensive insight into this topic, this survey reviews jailbreak and defense in multimodal generative models. First, given the generalized lifecycle of multimodal jailbreak, we systematically explore attacks and corresponding defense strategies across four levels: input, encoder, generator, and output. Based on this analysis, we present a detailed taxonomy of attack methods, defense mechanisms, and evaluation frameworks specific to multimodal generative models. Additionally, we cover a wide range of input-output configurations, including modalities such as Any-to-Text, Any-to-Vision, and Any-to-Any within generative systems. Finally, we highlight current research challenges and propose potential directions for future research. The open-source repository corresponding to this work can be found at https://github.com/liuxuannan/Awesome-Multimodal-Jailbreak.

I. INTRODUCTION

Multimodal generative models face jailbreak risks that exploit interactions among data types, while earlier reviews focused on narrower output modalities. This survey organizes attacks and defenses across a four-level lifecycle and covers varied modalities, model structures, and evaluation approaches.

I. INTRODUCTION

  • Multimodal systems create complex interaction spaces in which adversaries can exploit interactions among data types to bypass safeguards and produce inappropriate outputs.The introduction identifies this as a security and reliability concern as multimodal models become more widely deployed.Source: §I · p. 1
  • Earlier jailbreak reviews primarily covered Any-to-Text or Any-to-Image systems, leaving a unified framework across modalities and generative systems absent.The survey addresses this gap across text, images, audio, and video.Source: §I · pp. 1–2
  • The survey proposes a unified framework that categorizes multimodal jailbreak attacks and defenses at the input, encoder, generator, and output levels.It reviews attack methods and defense strategies across these four stages.Source: §I · p. 2
  • The survey reviews attack, defense, and evaluation strategies across input-output modalities and model structures.Its scope includes Any-to-Text, Any-to-Vision, and Any-to-Any configurations.Source: §I · p. 2
  • The survey discusses research limitations, challenges, and future directions for real-world applications.The paper is organized to cover preliminaries, attacks, defenses, evaluation, and future research opportunities.Source: §I · pp. 2–3

II. PRELIMINARIES

The preliminaries section introduces multimodal generative models and jailbreak to support comprehension of the survey.

II. PRELIMINARIES

  • The section provides a concise introduction to multimodal generative models and jailbreak for understanding the survey.Source: §II · p. 3

A. Multimodal Generative Models

The section distinguishes multimodal generative models by their input-output configurations and describes representative architectures for Any-to-Text, Any-to-Vision, and Any-to-Any systems.

A. Multimodal Generative Models

  • Any-to-Vision models condition diffusion-based visual generation on inputs from different modalities.For controllable image generation, a text encoder can interpret prompts while a vision processor estimates transition parameters during denoising.Source: A. Multimodal Generative Models · p. 3
  • Any-to-Text models encode multimodal inputs and connect them to an LLM through a modality interface to generate text.The interface may be a learnable projector or an expert model that translates non-text data into language.Source: A. Multimodal Generative Models · p. 3
  • Text-to-video models combine condition interpreters, frame-level vision processors, and temporal handlers that model motion and transitions.Source: A. Multimodal Generative Models · p. 3
  • Any-to-Any models either combine a language model with a separate generator or use a unified Transformer for multimodal understanding and generation.Next-GPT exemplifies the compositional approach, while Chameleon exemplifies a unified architecture.Source: A. Multimodal Generative Models · p. 3
  • Emu3 uses next-token prediction over unified discrete text, image, and video tokens in a single Transformer trained on multimodal sequences.Source: A. Multimodal Generative Models · p. 4

B. Jailbreak Attack •

The section frames jailbreak as crafting adversarial inputs that bypass safeguards to elicit harmful outputs, including through multimodal prompt, visual-input, and embedding vulnerabilities.

B. Jailbreak Attack •

  • Jailbreak attacks seek harmful or unintended outputs by crafting adversarial inputs that bypass a model’s safety mechanisms when direct malicious prompts are filtered.Source: B. Jailbreak Attack • · p. 4
  • The formulation seeks highly harmful generated content while keeping adversarial-prompt toxicity below a threshold to avoid detection and filtering.Sharm measures generated-content harmfulness; Stox measures manifest prompt toxicity.Source: B. Jailbreak Attack • · p. 4
  • Multimodal attackers can exploit textual prompts, visual inputs, and multimodal embeddings, targeting interactions among modalities.The expanded attack surface presents challenges to model safety and robustness.Source: B. Jailbreak Attack • · p. 4

C. Jailbreak Defense

The section describes how multimodal inputs complicate safety alignment and expose vulnerabilities, motivating defenses that protect security while preserving utility.

C. Jailbreak Defense

  • LLM safety alignment is improved through higher-quality data during SFT and alignment with human safety preferences during RLHF.The passage notes progress in LLM safety alignment while distinguishing the two stages’ roles.Source: C. Jailbreak Defense · p. 4
  • Additional modalities add complexity and expose risks, including unsafe embeddings introduced by modality encoders.The passage identifies encoder-related risks as a vulnerability specific to multimodal systems.Source: C. Jailbreak Defense · p. 4
  • Unsafe embeddings can propagate harmful content across modalities, motivating advanced defenses that preserve generative utility.The stated defense goal is ethical and secure use while preserving utility across applications.Source: C. Jailbreak Defense · p. 4

III. JAILBREAK ATTACK

The section classifies multimodal jailbreak attacks by model access: black-box attacks operate at the input or output surface, while gray- and white-box attacks target model components.

III. JAILBREAK ATTACK

  • The section categorizes multimodal jailbreak attacks as black-box, gray-box, or white-box.The categorization is organized around attacker access to the model.Source: §III · p. 4
  • In black-box settings, attackers cannot access the model and are limited to surface-level input and/or output interactions.Source: §III · p. 4
  • Gray- and white-box attacks are model-level attacks targeting the encoder and generator.Source: §III · p. 4

A. Black-box Jailbreak

This section surveys multimodal jailbreak attacks by attacker access and operation level, including black-box input and output manipulation, and reviews defenses that detect threats or steer generation toward safe responses. It distinguishes discriminative defenses from transformative ones and notes that threat classifiers can falsely flag benign inputs.

A. Black-box Jailbreak

  • Black-box attacks exploit observable input-output behavior, using crafted inputs or iterative queries to bypass safeguards and elicit prohibited responses.Attackers lack access to the target model’s internal architecture, parameters, or gradients.Source: A. Black-box Jailbreak · p. 4, §A.1 · p. 4

A.1 Input-level Attack

  • Input-level prompt engineering uses LLMs or trained proxies to refine jailbreak prompts, while agent-based methods decompose harmful intents into benign-seeming image descriptions.Examples include AutoJailbreak, Arondight, AdvWeb [64], DACA [69], UPAM, and BPSA [52].Source: §A.1 · pp. 5–6
  • Transfer-injection attacks embed malicious prompts in images or audio, using typography, generated scenarios, role-play imagery, or letter-splitting to obscure harmful instructions.Examples include MM-SafetyBench, Logical jailbreak [72], Visual-RolePlay [73], and AIAH [74].Source: §A.1 · p. 6

A.2 Output-level Attack

  • Multi-turn attacks progressively intensify harmful requests through dialogue, using self-adversarial prompting, victim-response feedback, generated-image feedback, or fictional role-play.Examples include the Safety Snowball Effect [80], IDEATOR [81], CoJ, Atlas [85], and VoiceJailbreak [21].Source: §A.2 · pp. 6–7

B.1 Encoder-level Attack

  • Encoder-level attacks optimize adversarial inputs to align embeddings with harmful content, using gradients when available or search over exposed embeddings otherwise.Jailbreak in Pieces and Redteaming Attack [86] optimize image embeddings; MMA-Diffusion and JPA [54] modify prompts, while Ring-A-Bell uses a genetic algorithm.Source: §B.1 · p. 7

B.2 Generator-level Attack

  • With full generator access, gradient-based attacks optimize toward adversarial goals using target responses, auxiliary diffusion models, target images, or classifier signals.Examples include visual attacks on IT2T models –[95], T2I attacks guided by auxiliary models or target images –, and image-editing attacks using MMA-Diffusion.Source: §B.2 · pp. 7–8

A. Discriminative Defense

  • Discriminative defenses classify threats using input statistics, encoder embeddings, generator activations, or response discrepancies, preserving the generation pipeline; input checks use statistical features, while encoder checks perturb inputs and inspect embeddings.CIDER [104] compares text and denoised-image embeddings and rejects inputs when the similarity difference exceeds a threshold.Source: §IV · p. 8, A. Discriminative Defense · pp. 8–9, §A.1 · p. 9

A.2 Encoder-level Defense; A.3 Generator-level Defense

  • Latent Guard maps blacklisted concepts and prompts into a shared latent space, GuardT2I reveals intent for blacklist and similarity checks, and NEARSIDE [107] classifies adversarial inputs by thresholding projections.NEARSIDE identifies the attack direction from hidden states by calculating the average embedding difference between benign and adversarial inputs; it classifies inputs as adversarial when projection onto that direction exceeds a defined threshold.Source: §A.2 · p. 10, §A.3 · p. 10

A.4 Output-level Defense

  • Output-level black-box defenses detect attacks through response divergence after input mutation or filter generated content with toxicity classifiers.JailGuard [108] uses 18 mutators and semantic response divergence; Espresso [109] applies a fine-tuned CLIP classifier to generated images.Source: §A.4 · p. 10

B.1 Input-level Defense

  • Input-level transformative defenses append tailored safety prompts or optimized guardrails, while refiners rewrite adversarial prompts and purify images before generation.AdaShield [110] selects a defense prefix by semantic similarity, UNIGUARD [111] optimizes modality-specific guardrails, and BlueSuffix [112] combines textual and visual purifiers with a suffix generator; POSI uses PPO to turn toxic prompts into clean ones.Source: §B.1 · p. 10

B.2 Encoder-level Defense

  • Encoder-level transformative defenses modify encoder behavior through training-phase fine-tuning or inference-phase guidance and refinement; fine-tuning seeks safety alignment in latent space to prevent harmful content propagating downstream.Sim-CLIP methods [59], [60] fine-tune the built-in vision encoder adversarially to enhance robustness against adversarial attacks; AdvUnlearn regularizes the trade-off between concept-erasure robustness and utility, while Safe-CLIP fine-tunes on LLM-generated synthetic data.Source: §B.2 · pp. 10–11

Transformative Defense

  • Encoder guidance adds learned latent vectors to prompt embeddings, while refiners identify and sanitize high-risk tokens while preserving benign ones.Self-discovery learns a concept-representing vector without labeled data or external models; ES uses internal token-level harmfulness scores for adaptive sanitization.Source: §B.2 · p. 11

B.3 Generator-level Defense

  • Generator tuning aligns outputs with safe responses and unlearns harmful information while seeking to preserve benign utility.VL-Guard uses paired harmful and safe images with instructions; SafeVLM [118] adds safety modules, BaThe [119] uses trainable soft embeddings, and Unlearn [120] combines harmful-output reduction with helpfulness and benign-performance objectives.Source: §B.3 · pp. 11–12
  • Generator defenses also erase unsafe image concepts through model tuning or attention edits, steer inference activations or denoising away from unsafe content, and prune concept-linked parameters.Methods include score- and attention-based concept erasure, reinforcement learning, steering vectors for multimodal inputs, and selective removal of concept-correlated neurons.Source: §B.3 · p. 12

B.4 Output-level Defense

  • Output-level transformative defenses adjust decoding or edit generated responses and images to remove unsafe content.CoCA adjusts token probabilities using a safety delta, IMMUNE [144] scores responses by safety, MLLM-Protector detects and detoxifies harmful text, ECSO uses query-aware image-to-text conversion before safe response generation, and LMIVS detects and localizes immoral image attributes before ethical editing.Source: §B.4 · p. 13

V. EVALUATION

This section reviews datasets and open-ended evaluation methods and metrics used in multimodal jailbreak evaluation.

  • Evaluation methods provide a standardized basis for comparing multimodal jailbreak attacks and defenses.Source: §V · p. 13
  • The review covers datasets relevant to jailbreak scenarios, followed by open-ended evaluation methods and metrics.Source: §V · p. 13

A. Evaluation Dataset

Evaluation datasets are grouped by whether they simulate malicious scenarios or are deliberately constructed to represent them.

  • The review classifies multimodal jailbreak datasets as simulated-malicious or real-malicious according to their construction intent.Source: A. Evaluation Dataset · p. 13
  • Simulated-malicious datasets typically repurpose existing benign datasets not designed for jailbreak scenarios.Source: A. Evaluation Dataset · p. 13
  • Real-malicious datasets are deliberately designed to cover a wide range of malicious topics and simulate real-world scenarios.Source: A. Evaluation Dataset · p. 13

A.1 Simulated-malicious Dataset

Simulated-malicious evaluation uses benign datasets as proxies for unsafe content, avoiding the creation of real-world malicious imagery.

  • Prior T2I studies use selected categories from benign datasets as proxies for unsafe classes to avoid generating real-world malicious imagery during evaluation.Source: §A.1 · p. 13
  • SneakyPrompt's Dog/Cat-100 contains 100 GPT-generated prompts describing dog or cat scenarios.Source: §A.1 · p. 13
  • UPAM designates 10 Microsoft COCO classes, including boat, bird, and clock, as harmful categories.Source: §A.1 · p. 13
  • UnlearnDiffAtk uses WikiArt and Imagenette samples for stylization and object-manipulation attacks.Source: §A.1 · p. 13

A.2 Real-malicious Dataset

The section surveys real-malicious datasets across generation and understanding tasks, then outlines evaluation approaches for open-ended model outputs.

A.2 Real-malicious Dataset

  • Text jailbreak resources include AdvBench's harmful strings and behaviors, RedTeam-2K's 2,000 queries, and HarmBench's 510 harmful behaviors.Source: §A.2 · p. 13
  • Adversarial image-text datasets support defense evaluation; Figstep, HADES, and JailBreakV-28K pair malicious prompts with typographic, synthesized, natural, blank, noise, or adversarial images.Source: §A.2 · pp. 13–14
  • MM-SafetyBench combines GPT-4-generated malicious questions and extracted unsafe phrases with synthesized and typography images.Source: §A.2 · p. 14
  • T2I/T2V datasets include NSFW-200's 200 ChatGPT-generated prompts, MMA's 1,000 harmful LAION-COCO captions, and VBCDE's 100 cases across five categories.Source: §A.2 · p. 14
  • MPUP contains 1,200 unsafe prompts across hate speech, physical harm, and fraud; I2P pairs 4,703 real-world harmful prompts with images across seven themes.Source: §A.2 · p. 14
  • I2P-based defense evaluations and specialized datasets include Unsafe Diffusion's 1,434 prompts, MACE-Celebrity's 200 names, and MACE-Art's 200 artists.Source: §A.2 · p. 14
  • T2VSafetyBench covers 12 safety aspects with 4,400 malicious prompts, while IT2T attacks combine text jailbreak datasets with real-world, generated, or adversarial images.Source: §A.2 · pp. 13–14

B. Evaluation Method; B.1 Manual Evaluation

B.2 Automatic Evaluation

The survey describes automated toxicity evaluation and robustness and utility metrics, then identifies underexplored multimodal attack and defense directions.

B.2 Automatic Evaluation

  • Automated toxicity evaluation includes detector-, GPT-, and rule-based approaches; detectors use pretrained classifiers trained on annotated datasets covering unsafe categories.The cited table presents toxicity detectors used for automatic evaluation of multimodal generative models.Source: §B.2 · p. 15, Table VI · p. 15

C. Evaluation Metric

  • Evaluation metrics divide into robustness measures of resistance to jailbreak inputs and utility measures of generated-content relevance, accuracy, and quality.Source: C. Evaluation Metric · p. 16

C.1 Robustness Metrics

  • ASR measures successful attacks among attempts: higher ASR indicates safeguard bypass, while reduced ASR suggests a defense prevents unsafe generation.Source: §C.1 · p. 16

C.2 Utility Metrics

  • Utility measures include prompt perplexity for text fluency, FID for generated-image quality, and CLIP Score for image-to-intent alignment.Lower FID indicates image features closer to real images; CLIP Score uses cosine similarity between adversarial images and clean prompts modified from toxic content.Source: §C.2 · p. 16

A. Multimodal Jailbreak Attack •

  • Video and audio jailbreak vulnerabilities remain relatively underexplored, with temporal dependencies and acoustic features posing distinct attack and defense challenges.Source: A. Multimodal Jailbreak Attack • · p. 16
  • Future attack research could jointly optimize modalities and examine combined Any-to-Any outputs, which may amplify harmful content’s credibility and psychological influence.One cited direction jointly optimizes textual and visual prompts rather than treating modalities in isolation.Source: A. Multimodal Jailbreak Attack • · pp. 16–17

B. Multimodal Jailbreak Defense •

  • Hybrid defenses are proposed because fine-tuning-only approaches may adapt poorly to evolving attacks, while training-free methods may lack effectiveness and accuracy.Other directions include preserving generation quality during concept erasure, improving defense transparency, and dynamically adjusting security levels for different application needs.Source: B. Multimodal Jailbreak Defense • · p. 17

C. Multimodal Jailbreak Evaluation •

The evaluation discussion identifies limits in detector coverage and attack-method coverage; the conclusion surveys attacks and defenses across modalities, architectures, and four lifecycle levels.

C. Multimodal Jailbreak Evaluation •

VII. CONCLUSION

  • The survey categorizes attacks and defenses across four levels and considers text, image, audio, and video modalities in Any-to-Text, Any-to-Vision, and Any-to-Any architectures.It also compares evaluation datasets, methods, and metrics and identifies challenges and future directions.Source: §VII · p. 17
Loading 2411.09259v2…