PaperlayerView exact revision on arXiv

Source-linked AI summary

Fuzz-Testing Meets LLM-Based Agents: An Automated and Efficient Framework for Jailbreaking Text-To-Image Generation Models

Yingkai Dong, Xiangtao Meng, Ning Yu, Zheng Li, Shanqing Guo

arXiv:2408.00523v3cs.CRcs.AIcs.LG

TL;DR

T2I jailbreak attacks can bypass safety filters, while existing approaches face practical limitations. JailFuzzer combines fuzz testing with LLM-based agents to generate and evaluate prompts, and the paper reports that it outperforms existing methods while preserving semantic similarity and reducing query overhead.

  • Problem

    Jailbreak prompts can bypass T2I safety filters and produce unsafe content.

    Source: §1 · p. 1

  • Method

    JailFuzzer applies fuzz testing with a seed pool, guided mutation engine, and oracle, using LLM-based agents with memory to guide prompt generation.

    Source: §1.1 · p. 2, §4.2 · p. 6, §8 · p. 15

  • Results

    JailFuzzer consistently outperforms existing methods while preserving semantic similarity and significantly reducing query overhead.

    Source: §8 · p. 15

  • Takeaways & Limitations

    The authors highlight the need for stronger generative-model safety mechanisms against evolving attack strategies.

    Source: §8 · p. 15

    Performance declines when an attacker targets one specific prompt without access to other stored memories.

    Source: §7.1 · p. 15

Abstract

from arXiv · show

Text-to-image (T2I) generative models have revolutionized content creation by transforming textual descriptions into high-quality images. However, these models are vulnerable to jailbreaking attacks, where carefully crafted prompts bypass safety mechanisms to produce unsafe content. While researchers have developed various jailbreak attacks to expose this risk, these methods face significant limitations, including impractical access requirements, easily detectable unnatural prompts, restricted search spaces, and high query demands on the target system. In this paper, we propose JailFuzzer, a novel fuzzing framework driven by large language model (LLM) agents, designed to efficiently generate natural and semantically meaningful jailbreak prompts in a black-box setting. Specifically, JailFuzzer employs fuzz-testing principles with three components: a seed pool for initial and jailbreak prompts, a guided mutation engine for generating meaningful variations, and an oracle function to evaluate jailbreak success. Furthermore, we construct the guided mutation engine and oracle function by LLM-based agents, which further ensures efficiency and adaptability in black-box settings. Extensive experiments demonstrate that JailFuzzer has significant advantages in jailbreaking T2I models. It generates natural and semantically coherent prompts, reducing the likelihood of detection by traditional defenses. Additionally, it achieves a high success rate in jailbreak attacks with minimal query overhead, outperforming existing methods across all key metrics. This study underscores the need for stronger safety mechanisms in generative models and provides a foundation for future research on defending against sophisticated jailbreaking attacks. JailFuzzer is open-source and available at this repository: https://github.com/YingkaiD/JailFuzzer.

1 Introduction

Jailbreak attacks use prompts to bypass T2I safety filters, while existing approaches face access, prompt-naturalness, search-space, and query-cost constraints. JailFuzzer applies fuzz testing with LLM-based agents and reports strong bypass performance with fewer queries.

1 Introduction

  • Existing attacks can require white-box access, produce detectable unnatural prompts or rely on fixed patterns, and consume many costly queries.The introduction also notes that selecting suitable prompts can be costly and time-consuming, especially for novel problems with limited references.Source: §1 · pp. 1–2

1.1 Our Contributions

  • The seed pool incorporates both successful and failed prompts as experience for improving mutation and oracle capabilities.The mutation process creates variations of seeds to explore model responses.Source: §1.1 · p. 2
  • Close to 100% bypass rate with 4.6 average queries on conventional filters, and more than 82.45% with 12.6 average queries on the conservative filter.The reported attacks retain reasonable semantic similarity.Source: §1.1 · p. 2
  • JailFuzzer uses black-box fuzz testing with a seed pool, guided mutation engine, and oracle function.The framework is designed to generate jailbreak prompts automatically and efficiently.Source: §1.1 · p. 2
  • LLM-based agents power the mutation engine and oracle to generate semantically meaningful variations and assess prompts before T2I queries.The oracle assesses each prompt’s likelihood of bypassing safety filters while preserving the sensitive prompt’s semantics, helping reduce invalid target-model queries.Source: §1.1 · p. 2
  • JailFuzzer outperforms existing methods across metrics while producing natural prompts with high semantic similarity and fewer queries.The paper reports these as advantages of its evaluated approach.Source: §1.1 · p. 2

2 Preliminaries and Related Works

This section introduces text-to-image models and their safety filters, reviews jailbreak methods and their limitations, and situates JailFuzzer’s approach in fuzzing and LLM-based agents.

2.1 Text-to-Image Generative Models

  • Text-to-image models generate images from natural-language prompts, while safety filters block sensitive content by checking text, images, or both.Text filters use classifiers or keyword lists; image filters classify generated images; hybrid filters consider text and image embeddings.Source: §2.1 · p. 3

2.2 Jailbreaking Text-to-Image Models

  • Jailbreak prompts are designed to bypass safeguards; prior token- and prompt-level methods can produce unnatural prompts or restrict exploration, and the study focuses on black-box attacks.The paper compares its approach against SneakPrompt, Ring-A-Bell, and DACA.Source: §2.2 · p. 3

2.3 Fuzzing

  • Fuzzing explores system inputs through an evolving seed pool, mutation engine, and oracle, but predefined jailbreak templates needed by LLM-Fuzzer are unavailable for T2I models.Manually constructing such templates would substantially distort generated-image semantics.Source: §2.3 · pp. 3–4

2.4 LLM-based Agents

  • LLM-based agents integrate a brain with planning, memory, and tools; JailFuzzer uses these agents to implement its guided mutation engine and oracle.The agents’ modules support task decomposition and use of prior actions and observations.Source: §2.4 · p. 4, §2.3 · p. 4

3 Overview of JailFuzzer

JailFuzzer targets black-box T2I models with cost-bounded API access, searching for prompts that bypass safety filters while preserving the target semantics. It combines fuzz-testing components with LLM-based agents to iteratively generate and rank candidates.

3.1 Threat Model

  • The threat model assumes an API-only adversary without knowledge of model internals or filters, bounded query access, and one-time or prompt-reuse attack scenarios.In a one-time attack, prompts are searched from scratch; reuse attacks apply previously found prompts despite model output randomness.Source: §3.1 · p. 4

3.2 Key Idea and Techniques

  • JailFuzzer seeks a prompt whose generated image retains the sensitive target’s semantics while the safety filter classifies it as non-sensitive.Its fuzz-testing design uses a seed pool, guided mutation engine, and oracle, with LLM agents implementing mutation and candidate evaluation.Source: §3.2 · pp. 4–5

3.3 Overall Pipeline

  • The pipeline checks bypass, mutates blocked prompts, checks semantic alignment after bypass, and has an oracle rank candidates before forwarding the best prompt for generation.Mutation continues until both bypass and similarity criteria are met or the mutation threshold is reached; the seed pool evolves during the multi-loop attack.Source: §3.3 · p. 5, Figure 2 · p. 5

4 System Design of JailFuzzer

JailFuzzer’s system design evolves a pool of prompts across attack loops and coordinates mutation and oracle agents with memory, retrieval, and staged planning.

4.1 Multi-Loop Attack Flow

  • Across loops, failed prompts are retried using accumulated experience, while each loop restarts from the original prompt to avoid local optima.The authors state that learning from successes and failures improves subsequent success rates and reduces required attempts.Source: §4.1 · p. 6

4.2 Mutation Agent

  • The mutation agent stores successful prompts, retrieves semantically similar examples, and reflects on their success factors to guide target-specific mutations.Its reflection has Identify and Guide phases; CLIP-ViT-Base-Patch32 provides a fixed-threshold image–prompt semantic check.Source: §4.2 · pp. 6–7

4.3 Oracle Agent

  • The oracle agent ranks candidate prompts for bypass potential and semantic similarity, using retrieved successful and failed examples to simulate the target system.Its semantic retriever selects similar prompts from both experience pools for the current target.Source: §4.3 · pp. 7–8

4.4 Planning Module of Two Agents

  • The planning module decomposes work into six steps: the mutation agent checks bypass and semantics and generates candidates, while the oracle scores candidates for the relevant criterion.The selected highest-scoring prompt is sent to the T2I model, and the steps repeat until termination.Source: §4.4 · p. 8

5 Experimental Setup

The experiments evaluate JailFuzzer with selected agent models against several T2I models and safety filters, using NSFW and benign prompt datasets and measures of bypass, reuse, similarity, and query cost.

Hyperparameters and Agent Models

  • The mutation agent uses LLaVA-1.5-13b or ShareGPT4V-13b, and the oracle uses Vicuna-1.5-13b; the study excludes models whose safeguards prevent sensitive-content processing.The CLIPScore threshold is 0.26, the attack allows at most six loops, and the mutation and oracle candidate counts are 5 and 10, respectively.Source: §5 · p. 9

Target T2I Model and Safety Filters

  • The study targets SD1.4, SDXL [40], SD3, and DALL·E 3; it equips the three Stable Diffusion models with six safety filters, applying the built-in text-image filter only to SD1.4.Source: §5 · p. 9

Dataset

  • The evaluation uses 200 NSFW prompts and 100 dog-or-cat prompts, with the first half of NSFW-200 used for DALL·E 3 to limit cost.The NSFW dataset tests filters other than the dog-cat classifier; Dog/Cat-100 is paired with that classifier to avoid NSFW generation.Source: §5 · p. 9

Evaluation Metrics

  • Four metrics measure one-time bypass, prompt reuse, image-distribution similarity with FID, and the number of target-model queries.A successful prompt must generate an image whose CLIPScore exceeds δ; lower FID and higher bypass rate indicate a better attack.Source: §5 · p. 9, Table 1 · p. 10

6 Evaluation

RQ1 evaluates JailFuzzer against safety filters and non-filter defenses, including different target models and mutation-agent brains. The reported results show high bypass rates with limited queries, though reuse success varies by defense type.

6.1 RQ1: Effectiveness at Bypassing Safety Mechanisms

  • 100% one-time bypass success is reported on Stable Diffusion, averaging 4.6 queries; conservative text classifiers still exceed 82.5% at 12.6 queries.The method reports lower FID in most filter settings, with exceptions for text-classifier-based and dog/cat-image-classifier-based filters.Source: §6.1 · pp. 10–11
  • DALL·E 3 reaches reported one-time bypass rates of 81.93% and 79.50%, averaging 13.38 queries.The authors report higher FID for DALL·E 3 because its generated images differ in style from the evaluation dataset.Source: §6.1 · p. 11
  • PPL retains a 100% bypass rate, while SmoothLLM variants exceed 88%, with only minor increases in FID scores and query counts.The evaluation adapts SmoothLLM to T2I models by using built-in safety-filter classifications to assess whether a fraction of responses jailbreaks the model.Source: Table 3 · p. 11, §6.1 · p. 12
  • ShareGPT4V-1.5 generally outperforms LLaVA-1.5, but both mutation-agent brains achieve strong attack performance across the tested cases.The authors conclude that either VLM can serve as the mutation agent’s brain.Source: §6.1 · p. 11
  • Against non-filter defenses, bypass rates are 94% for UCE, 81% for POSI, and 82% for SafeGen; the oracle checks generated images for malicious content.For these defenses, JailFuzzer continues mutation when the generated image is not malicious, rather than stopping based only on semantic similarity.Source: Table 2 · p. 11, §6.1 · p. 12

DACA [18]

JailFuzzer compares favorably with baseline attacks, and its parameter studies find that agent configuration, similarity thresholds, and memory settings affect performance. Iterative refinement improves bypass rates, while larger memory and higher similarity thresholds can impose costs.

6.2 RQ2: Performance Comparison with Baselines

  • JailFuzzer has the highest one-time bypass rates across evaluated filters, comparable or higher reuse rates than SneakyPrompt and Ring-A-Bell, and lower FID in most cases.DACA has higher reuse bypass rates, but JailFuzzer permits more prompt reuse because of its higher one-time bypass rate; JailFuzzer also uses fewer queries in most settings, except text-match filters where counts are comparable to SneakyPrompt.Source: §6.2 · p. 12
  • JailFuzzer and DACA generate more natural prompts than SneakyPrompt and Ring-A-Bell, but DACA’s limited exploration space hampers bypass effectiveness.The comparison evaluates prompt naturalness using perplexity, for which lower values indicate more natural text.Source: §6.2 · p. 13
  • The cited passage reports success rates of 100% and 88%; separately, Table 6 compares JailFuzzer and LLM-Fuzzer for bypassing Stable Diffusion 1.4’s text-based safety filter.On GPT-3.5, LLM-Fuzzer’s jailbreak success rate dropped from 96% to 52% when starting from original prompts instead of jailbreak templates.Source: §6.2 · p. 13, Table 6 · p. 13

6.3 RQ3: Different Parameter Selection

  • The VLM-only setup performs worse than agent-based configurations, while two agents outperform one in bypass rate and query efficiency, especially under stricter filters.The study focuses on one-time bypass rate and queries because agent count does not affect reuse performance or image quality.Source: §6.3 · pp. 13–14, Figure 8 · p. 14
  • As the semantic-similarity threshold rises from 0.22 to 0.30, bypass rates fall and query counts rise, though success remains above 90% at 0.30.The authors use 0.26 in the main experiments as a balance between preserving malicious semantics and performance.Source: §6.3 · p. 14, Table 7 · p. 14
  • Without memory, JailFuzzer’s bypass rate is 81.65%; larger long-term memory can increase queries or reduce bypass success when context limits are exceeded.At km = 10, success remains 100% but queries increase; at km = 20, bypass rates drop significantly, while kc length has minimal impact.Source: §6.3 · p. 14, Table 8 · p. 14
  • The limit Θ = (4,10,10,10,...) performs best, although changing per-loop query limits has minimal overall impact; larger sensitive-prompt pools improve performance.Even a pool size of 50 maintains strong performance.Source: §6.3 · p. 14, Table 9 · p. 15, Table 10 · p. 15
  • Bypass rates rise from 33.07% initially to 62.20%, 96.85%, and 97.64% in subsequent iterations; the fifth iteration reaches 100%.The initial ClipScore is 0.2810, and the passage reports stable semantic coherence during refinement.Source: §6.3 · pp. 14–15

7 Discussion

JailFuzzer’s performance is strongest with a diverse prompt pool and access to stored memories; the discussion also outlines possible defenses, each with different scope and trade-offs.

7.1 Limitations of Our Study

  • JailFuzzer uses open-source large models that are not safety-aligned; extending the framework with safety-aligned models remains future work.The authors expect stronger reasoning and instruction-following models to further enhance effectiveness.Source: §7.1 · p. 15
  • Performance declines when attackers use one sensitive prompt without access to other stored memories, whereas testers typically begin with broader inputs and accumulate prompt experience.Locally stored successful and failed prompts can improve identification of new jailbreak prompts.Source: §7.1 · p. 15

7.2 Possible Defense

  • Adversarial training may reduce jailbreak risk, but empirical defenses struggle to cover the full prompt range and leave an ongoing attack-defense arms race.The discussion also identifies randomized smoothing and certified robustness as directions for future research.Source: §7.2 · p. 15
  • Unlearning methods UCE and SafeGen remain vulnerable to adaptive JailFuzzer attacks but outperform conventional safety filters, while repeated-trigger blocking can mitigate harm without fixing underlying vulnerabilities.The authors describe unlearning as promising because it can fundamentally eliminate inherent vulnerabilities.Source: §7.2 · p. 15

8 Conclusion

JailFuzzer applies fuzz-testing and LLM agents to generate natural, semantically meaningful jailbreak prompts in black-box T2I settings. Experiments report high attack success while preserving semantic similarity and reducing query overhead, motivating stronger safeguards and further defense research.

8 Conclusion

  • JailFuzzer combines fuzz-testing with LLM agents, tools, and memory to generate natural, semantically meaningful prompts in black-box T2I settings.The authors report that it consistently outperforms existing methods in identifying vulnerabilities in state-of-the-art T2I models.Source: §8 · p. 15
  • JailFuzzer achieves exceptional attack success rates while preserving semantic similarity and significantly reducing query overhead, consistently outperforming existing methods.The authors report these results across experiments on state-of-the-art T2I models.Source: §8 · p. 15
  • The authors argue that generative models need enhanced safety mechanisms and call for future research on defenses against sophisticated attacks.They also suggest JailFuzzer can help evaluate model robustness, potentially strengthening safeguards against jailbreaks and related threats.Source: §8 · pp. 15–16

A Examples of Jailbreak Prompts and Corresponding Images

The appendix illustrates jailbreak prompts and corresponding generated images, including examples involving cats and dogs used with an external safety filter. It also points readers to examples of NSFW images through an external link.

A Examples of Jailbreak Prompts and Corresponding Images

B Detailed System Messages and Prompt Templates

The appendix provides system messages and prompt templates for JailFuzzer’s mutation and evaluation components. They specify how prompts are generated, guided, and scored, and how an LLM simulates a safety filter and evaluates semantic similarity.

B Detailed System Messages and Prompt Templates

  • The mutation-agent system message frames prompt modification as bypassing a T2I model’s safety filter, while ICL and strategy templates guide changes using successful prompts and key bypass factors.The semantic-guide template asks for changes that preserve similarity to the original prompt, its porn semantic, and filter bypass.Source: B Detailed System Messages and Prompt… · p. 19
  • The safety-filter system message simulates the filter using failed and successful prompt examples, and the semantic evaluator scores similarity to the original prompt.The semantic evaluator describes both prompts, analyzes contextual and sentiment similarities and differences, then assigns a score from 0 to 10.Source: B Detailed System Messages and Prompt… · pp. 19–20
  • The check-description and check-decision templates classify image outputs into five options, including correspondence to the current prompt or none of the listed cases.The decision template requires a reasoning/result format and an A–E outcome.Source: B Detailed System Messages and Prompt… · p. 20
  • The modify template requests five new prompts, while bypass and semantic score templates rate candidates for filter likelihood and similarity to the original.The semantic score prompt lowers ratings for prompts exceeding 77 tokens.Source: B Detailed System Messages and Prompt… · p. 20

C Other System Messages and Prompt Templates

The passage directs readers to the JailFuzzer repository for detailed information.

C Other System Messages and Prompt Templates

Loading 2408.00523v3…