PaperlayerView exact revision on arXiv

Source-linked AI summary

SneakyPrompt: Jailbreaking Text-to-image Generative Models

Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, Yinzhi Cao

arXiv:2305.12082v3cs.LG

TL;DR

Text-to-image safety filters are intended to prevent sensitive-image generation, but their robustness to adversarial prompt manipulation remains unknown. SneakyPrompt automates token replacement with reinforcement-learning guidance and successfully jailbreaks closed-box DALL·E 2 and open-source Stable Diffusion filters, outperforming existing text-based attacks.

  • Problem

    The paper addresses the limited evidence about whether practical text-to-image safety filters withstand adversarial prompt manipulations that seek NSFW generations.

  • Method

    SneakyPrompt repeatedly queries a target text-to-image model and uses reinforcement learning to replace filtered tokens while rewarding semantic similarity and safety-filter bypass.

  • Results

    SneakyPrompt successfully generates NSFW images through DALL·E 2’s closed-box filter and bypasses diverse Stable Diffusion filters while outperforming existing text-based attacks.

  • Takeaways & Limitations

    The results indicate that existing text-to-image guardrails are insufficient and motivate new guardrails to limit societal harms.

  • Takeaways & Limitations

    The threat model assumes bounded target-model queries and access to a local shadow text encoder.

Abstract

from arXiv · show

Text-to-image generative models such as Stable Diffusion and DALL$\cdot$E raise many ethical concerns due to the generation of harmful images such as Not-Safe-for-Work (NSFW) ones. To address these ethical concerns, safety filters are often adopted to prevent the generation of NSFW images. In this work, we propose SneakyPrompt, the first automated attack framework, to jailbreak text-to-image generative models such that they generate NSFW images even if safety filters are adopted. Given a prompt that is blocked by a safety filter, SneakyPrompt repeatedly queries the text-to-image generative model and strategically perturbs tokens in the prompt based on the query results to bypass the safety filter. Specifically, SneakyPrompt utilizes reinforcement learning to guide the perturbation of tokens. Our evaluation shows that SneakyPrompt successfully jailbreaks DALL$\cdot$E 2 with closed-box safety filters to generate NSFW images. Moreover, we also deploy several state-of-the-art, open-source safety filters on a Stable Diffusion model. Our evaluation shows that SneakyPrompt not only successfully generates NSFW images, but also outperforms existing text adversarial attacks when extended to jailbreak text-to-image generative models, in terms of both the number of queries and qualities of the generated NSFW images. SneakyPrompt is open-source and available at this repository: \url{https://github.com/Yuchen413/text2image_safety}.

1. Introduction

Text-to-image models can generate sensitive images despite safety filters, whose robustness to adversarial prompt manipulation remains unknown. SneakyPrompt addresses this gap with automated token perturbation guided by reinforcement learning and successfully bypasses filters on DALL·E 2 and Stable Diffusion.

  • Text-to-image models have broad applications but may generate sensitive NSFW content, motivating the use of safety filters.
  • The robustness of practical safety filters to adversarial prompt manipulation remains unknown.
  • Existing text-based adversarial attacks target classification decisions rather than preserving NSFW image generation and fail against DALL·E 2’s closed-box filter in evaluation.
  • SneakyPrompt searches for alternative tokens that preserve prompt semantics and generated NSFW images while bypassing the filter.
  • Reinforcement learning guides perturbations using semantic-similarity and filter-bypass rewards to reduce target-model queries.
  • SneakyPrompt successfully finds adversarial prompts that make DALL·E 2 generate NSFW images despite its closed-box safety filter.
  • SneakyPrompt bypasses diverse open-source Stable Diffusion safety filters and outperforms existing text-based adversarial attacks.

2. Related Work and Preliminary

Related work covers vulnerabilities and adversarial attacks on text-to-image and learning models, while highlighting why conventional text attacks are insufficient for safety-filter bypass. Reinforcement learning supplies feedback-driven search but introduces large action-space and reward-design challenges.

  • Prior work studies membership inference, image extraction, multilingual adversarial examples, and adversarial prompts against text-to-image models.
  • Adversarial-example research spans computer vision and NLP, including methods that perturb text while attempting to preserve meaning.
  • Existing attacks applied to text-to-image safety filters may lose NSFW semantics, require excessive queries, and produce prompts that are not reusable.
  • Reinforcement learning uses states, actions, policies, environments, and rewards to incorporate feedback into sequential decisions.
  • Applying reinforcement learning to adversarial prompts requires managing a large word-space action set and designing rewards that preserve generated-image NSFW semantics while bypassing filters.

3. Problem Formulation

The paper formalizes safety filters, adversarial prompts, and a closed-box threat model for text-to-image systems. It distinguishes filter types, requires both bypass and semantic preservation, and evaluates one-time and reusable attacks under query constraints.

  • 3.1. Definitions: Problem Formulation: The paper defines safety filters as mechanisms that prohibit images with sensitive adult, violent, or political content.
  • 3.1. Definitions: Problem Formulation: Safety filters are categorized as text-based, image-based, or text-image-based according to the spaces they inspect.
  • 3.1. Definitions: Problem Formulation: An adversarial prompt must bypass the filter and generate an image with visual semantics similar to those of the blocked target prompt.
  • 3.1. Definitions: Problem Formulation: Figure 2 illustrates adversarial prompts whose generated cat or dog images preserve the target semantics after an external filter blocks the original prompts.
  • 3.2. Threat Model: Threat Model: The adversary has closed-box query access to the online model, receives filtered or generated outputs, and cannot access intermediate embeddings or diffusion gradients.
  • 3.2. Threat Model: Threat Model: The adversary may use a shadow text encoder that differs from the target encoder or exactly matches a publicly available encoder.
  • 3.2. Threat Model: Threat Model: The paper considers one-time attacks and re-use attacks, with re-use as the default because it avoids repeatedly querying the target model.

4. SneakyPrompt

SneakyPrompt searches for adversarial prompts that preserve target-image semantics while crossing a safety filter’s decision boundary. It compares heuristic token search with reinforcement-learning-guided search to reduce online queries.

  • Overview: The attack seeks prompts that preserve target semantics, bypass the safety filter, and minimize online queries.These are formalized as three objectives: semantic similarity, filter bypass, and query minimization.
  • Overview: SneakyPrompt replaces sensitive prompt tokens with non-sensitive alternatives while preserving the target prompt’s image semantics.The search constructs adversarial prompts from replacing tokens and evaluates similarity between generated-image and target-prompt embeddings.
  • Overview: The search space restricts each replacing token to vocabulary items whose lengths are at most l, avoiding search over the full vocabulary.The resulting space contains tuples of nm replacing tokens drawn from Dl.
  • Overview: The pipeline embeds the target prompt, samples replacing tokens, queries the target model, and repeats until bypass and similarity thresholds are met.A shadow text encoder supplies the target embedding, while the target model returns the safety-filter result and generated image.
  • Baseline Search with Heuristics: SneakyPrompt-base uses BruteForce, GreedySearch, or BeamSearch to sample replacing tokens as heuristic baselines.GreedySearch selects tokens closest to the target in shadow-encoder embedding space, while BeamSearch maintains k candidate lists.
  • Guided Search via Reinforcement Learning: SneakyPrompt-RL treats replacing-token sequences as actions and model responses as environment feedback for policy-guided search.Rewards reflect safety-filter bypass and semantic similarity; negative rewards reduce the likelihood of resampling unsuccessful token sequences.
  • Guided Search via Reinforcement Learning: The RL search can stop early when replacement expansion exceeds 0.3 or reward differences remain below 1e-4 across three consecutive queries.These stopping rules limit inefficient exploration.

5. Experimental Setup

The experiments evaluate SneakyPrompt on Stable Diffusion and DALL·E 2 using NSFW and non-NSFW prompt datasets, multiple safety-filter types, and attack-quality metrics.

  • Target Models: Experiments target Stable Diffusion’s open-source model and DALL·E 2 through OpenAI’s official online API.Stable Diffusion is the default target model, and experiments use two GeForce RTX 3090 graphics cards.
  • Prompt Datasets: The evaluation uses NSFW-200 with 200 NSFW prompts and Dog/Cat-100 with 100 dog-or-cat prompts.The dog/cat dataset tests bypassing safety filters without NSFW content that may make people uncomfortable.
  • Safety Filters: The safety-filter evaluation includes text-based, image-based, and text-image-based filters.The Stable Diffusion filter blocks images using cosine similarity between CLIP image embeddings and predefined sensitive-concept text embeddings.
  • Evaluation Metrics: Bypass rate measures successful bypasses, while FID assesses generated-image quality relative to reference image sets.Higher bypass rate and lower FID indicate better attack performance.
  • Evaluation Metrics: Online-query count measures the number of target-model queries used to search for an adversarial prompt and excludes reuse attacks.Reuse attacks require no additional adversarial-prompt-generation queries.

6. Evaluation

SneakyPrompt bypasses diverse text-to-image safety filters while preserving target-prompt semantics, including DALL·E 2’s closed-box filter. Its RL variant outperforms baseline attacks, but parameter choices require balancing bypass rate, image similarity, and query cost.

  • RQ1: Effectiveness at Bypassing Safety Filters: 96.37% average one-time bypass rate and 14.68 average queries were achieved across six Stable Diffusion safety filters.Four filters reached a 100.00% one-time bypass rate, with at least 2.26 queries required for one filter.
  • RQ1: Effectiveness at Bypassing Safety Filters: 57.15% one-time bypass rate was achieved against DALL·E 2’s closed-box safety filter with 24.49 average queries.The reported re-use bypass rate for DALL·E 2 was 100%.
  • RQ1: Effectiveness at Bypassing Safety Filters: Safety-filter scale correlated positively with robustness, while combined text-and-image filters outperformed filters relying on a single factor.Image-based filters also had lower re-use bypass rates because uncontrollable random seeds changed regenerated images.
  • RQ2: Performance Comparison with Baselines: SneakyPrompt-RL outperformed SneakyPrompt-base, existing text-based adversarial examples, and manual prompts across the reported baseline comparison.In one-time searches, SneakyPrompt-RL used 50% fewer queries than TextFooler, while re-use prompts benefited from image-semantic similarity guidance.
  • RQ3: Study of Different Parameter Selection: Cosine-similarity rewards produced higher bypass rates and better image semantic similarity, whereas ℓ2-distance rewards required fewer online queries.This comparison exposes a direct trade-off between attack effectiveness, semantic preservation, and query efficiency.
  • RQ4: Explanation of Bypassing: 0.298 target-image semantic similarity was close to 0.267–0.289 for images generated from adversarial prompts across different safety filters.These values indicate that the generated images retained the target-prompt semantics.

7. Conclusion, Discussion, and Future Work

SneakyPrompt shows that black-box safety filters can be bypassed with few queries, while motivating stronger guardrails and adversarially trained filters.

  • SneakyPrompt jailbreaks a black-box safety filter to produce an NSFW image with a small number of model queries.
  • Reinforcement learning strategically guides prompt perturbations by using query results, reducing the number of required queries.
  • The findings indicate that existing text-to-image guardrails are insufficient and that more robust safety filters are urgently needed.

Appendix A. Examples of Generated Sensitive Images

The appendix provides examples of sensitive generated images and outlines SneakyPrompt-RL’s inputs, search process, query loop, and stopping conditions.

  • Appendix A. Examples of Generated Sensitive Images: SneakyPrompt-RL takes a target prompt, text-to-image model, shadow text encoder, threshold, query budget, policy network, learning rate, and dataset as inputs.It outputs an adversarial prompt and a generated image when available.
  • Appendix A. Examples of Generated Sensitive Images: The algorithm initializes sensitive-token search and prompt embeddings, then samples token replacements until semantic similarity reaches the threshold before querying the target model.It expands the search space after five consecutive failed bypasses and stops when rewards stagnate or at least 30% of prompt tokens are replaced.
  • Appendix A. Examples of Generated Sensitive Images: Figure 7 presents generated images containing disturbing, violent, nudity, or sexual content.The examples are distributed through an external link requiring author-provided access.

B.1. Summary of Paper

The paper proposes SneakyPrompt to circumvent NSFW filters and reports a 57.15% bypass rate against DALL·E 2’s closed-box safety filter.

  • B.1. Summary of Paper: SneakyPrompt is an attack framework designed to circumvent NSFW content filters in generative text-to-image models.
  • B.1. Summary of Paper: 57.15% bypass rate is reported for the closed-box safety filter of DALL·E 2.
  • B.1. Summary of Paper: The evaluation demonstrates that the framework can bypass a closed-box text-to-image safety filter.

B.2. Scientific Contributions

The paper provides a valuable step forward in an established field and creates a new tool to enable future science.

  • The work provides a valuable step forward in an established field.
  • The work creates a new tool to enable future science.

B.3. Reasons for Acceptance

The paper was viewed as a step forward because it applies content-filter evasion to large text-to-image models with an RL-based attack algorithm. It also offers an open-source framework for future research on both open-source and closed-box NSFW filters.

  • SneakyPrompt is presented as the first RL-based attack algorithm capable of bypassing DALL·E 2’s closed-box safety filter.
  • The paper introduces an automated attack framework that can circumvent open-source and closed-box NSFW filters used by generative image models.

B.4. Noteworthy Concerns

The main concern is that the evidence for applying SneakyPrompt to closed-box safety filters is narrow. Only DALL·E 2 is evaluated as a closed-box filter.

  • Only DALL·E 2 is evaluated as a closed-box safety filter, limiting evidence for broader applicability.

Appendix C. Response to the Meta-Review

The authors agree that evaluating additional closed-box safety filters would strengthen the paper, but report that access constraints prevented such an evaluation. They identify improved interfaces, funding, and partnerships as possible routes for future work.

  • The authors agree that evaluating additional closed-box safety filters would strengthen the paper.
  • Poorly documented programming interfaces and high usage costs prevented evaluation of more closed-box safety filters.
  • Future work may pursue additional evaluations as access improves, funding becomes available, or partnerships facilitate broader system access.
Loading 2305.12082v3…