PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts

Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, Wei-Chen Chiu

arXiv:2309.06135v3cs.CLcs.CV

TL;DR

The paper addresses limited evidence about whether text-to-image safety mechanisms withstand diverse problematic prompts. It introduces P4D, which uses an unconstrained diffusion model and prompt engineering to automatically find bypassing prompts, and reports vulnerabilities across deployed protections, including a result that over 60% of prompts found in ESD transfer to other safe models.

  • Problem

    The reliability of text-to-image safety mechanisms against diversified problematic prompts remains largely unexplored despite concerns about copyrighted and NSFW outputs.

  • Method

    P4D uses prompt engineering and an unconstrained diffusion model to automatically find problematic prompts that bypass safety-equipped T2I models.

  • Results

    Over 60% of prompts found in ESD successfully jailbreak other safe T2I models, demonstrating cross-model transferability of discovered prompts.

  • Takeaways & Limitations

    P4D provides developers with a red-teaming tool for testing safety reliability and identifying challenging prompts for stronger mechanisms.

  • Takeaways & Limitations

    P4D requires access to both unconstrained and safe T2I models, which the authors describe as somewhat restrictive.

Abstract

from arXiv · show

Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown remarkable ability in high-quality content generation, and become one of the representatives for the recent wave of transformative AI. Nevertheless, such advance comes with an intensifying concern about the misuse of this generative technology, especially for producing copyrighted or NSFW (i.e. not safe for work) images. Although efforts have been made to filter inappropriate images/prompts or remove undesirable concepts/styles via model fine-tuning, the reliability of these safety mechanisms against diversified problematic prompts remains largely unexplored. In this work, we propose Prompting4Debugging (P4D) as a debugging and red-teaming tool that automatically finds problematic prompts for diffusion models to test the reliability of a deployed safety mechanism. We demonstrate the efficacy of our P4D tool in uncovering new vulnerabilities of SD models with safety mechanisms. Particularly, our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered "safe" can actually be manipulated to bypass many deployed safety mechanisms, including concept removal, negative prompt, and safety guidance. Our findings suggest that, without comprehensive testing, the evaluations on limited safe prompting benchmarks can lead to a false sense of safety for text-to-image models.

1. Introduction

Text-to-image diffusion models can generate inappropriate content, while existing safety mechanisms remain vulnerable to diverse problematic prompts. P4D uses prompt engineering and an unconstrained model to discover prompts that bypass deployed protections.

  • Internet-scale training data can cause models to generate copyrighted or NSFW content.
  • Existing safety mechanisms, including negative prompts, SLD, and ESD, are only partially effective against inappropriate image generation.These methods restrict text embeddings during inference or fine-tune the model.
  • P4D combines prompt engineering with an unconstrained diffusion model to automatically find problematic prompts for red-teaming safety-equipped T2I models.The discovered prompts can support model-misbehavior analysis and stronger safety mechanisms.
  • Around half of originally safe prompts in the I2P benchmark can be manipulated into problematic prompts by P4D.These prompts can evade existing safety mechanisms and produce inappropriate outputs.
  • Disabling safety mechanisms during debugging can make P4D more effective at finding prompts that later bypass those mechanisms during inference.The authors describe this as a potential information-obfuscation effect during red-teaming.

2. Related work

Prior work studies red-teaming for finding generative-model vulnerabilities, prompt engineering for adapting models through input prompts, and safety mechanisms for restricting inappropriate diffusion-model outputs. P4D combines these directions to test safety mechanisms at scale.

  • AI red-teaming tools: AI red-teaming searches for vulnerabilities and weaknesses to improve model security and stability.Recent approaches revise generated text or iteratively optimize user inputs toward model failures.
  • Prompt engineering: Prompt engineering adapts pretrained models by modifying input text with hard or soft prompts.Hard prompts use interpretable discrete tokens, whereas soft prompts use continuous-valued embeddings.
  • Prompt engineering: Soft prompts offer a larger search space but are often uninterpretable or non-transferable across language models.
  • Diffusion models with safety mechanisms: Diffusion-model safety mechanisms use guidance-based or fine-tuning-based methods to prevent inappropriate image generation.Guidance methods include negative prompts and SLD, while ESD represents fine-tuning-based protection.
  • Diffusion models with safety mechanisms: P4D addresses safety corner cases by identifying problematic prompts at scale for developers to test and mitigate.The discovered prompts can provide challenging cases for improving model robustness.

3. Background

Diffusion models generate data by progressively denoising noisy states, while latent diffusion performs this process in a compressed latent space. Conditioning the latent process on text enables efficient text-to-image generation.

  • Diffusion Models: Diffusion models learn generation through a backward denoising process that reverses forward steps adding Gaussian noise.
  • Diffusion Models: The diffusion training objective teaches a parameterized model to predict the noise added to intermediate noisy images.The time step t ranges from 1 to the maximum step T.
  • Latent Diffusion Models: Latent diffusion moves both forward and backward processes from pixel space into latent space to improve efficiency.An encoder and decoder provide the transformation between image and latent representations.
  • Latent Diffusion Models: Latent diffusion extends diffusion models to conditional generation by predicting noise from noisy latents, time steps, and text conditions.Training uses image–condition pairs.

4. Methdology

P4D red-teams safety-equipped text-to-image diffusion models by optimizing prompts that make a protected model reproduce inappropriate content generated by an unconstrained model. It combines latent-space noise-prediction matching with prompt projection and decoding to obtain interpretable hard prompts.

  • Framework overview: P4D uses an unconstrained diffusion model G and a safety-equipped model G′ to find prompts that bypass G′ for a target inappropriate concept or object.The optimized prompt makes G′ produce content resembling the output obtained from G.
  • Framework overview: The workflow first uses G to generate an inappropriate image, encodes it into a latent representation, and samples a noisy latent at an arbitrary diffusion timestep.The latent is obtained with G’s encoder before optimization against the protected model.
  • Prompt optimization: P4D optimizes a new or modified prompt by matching G’s and G′’s noise predictions from the same noisy latent, encouraging G′ to reconstruct similar inappropriate content.The optimization operates directly in latent space while both noise predictors remain fixed.
  • Prompt optimization: The method optimizes continuous token embeddings, projects them to nearest vocabulary embeddings, and decodes the result into legible text prompts.Projection is non-differentiable, so the continuous embeddings are updated using the loss gradient before hard-prompt decoding.
  • Prompt variants: P4D-N initializes a fixed number of tokens independently of the original prompt, whereas P4D-K inserts learnable tokens relative to the original prompt length.P4D-K addresses the concern that fixed-length P4D-N prompts may be insufficient for complex content.
  • Framework overview: Figure 2 summarizes P4D as prompt-engineering-based red-teaming of safety mechanisms including negative prompts, SLD, and ESD with help from an unconstrained model.The protected model is denoted G′ and the unconstrained model G.
  • Prompt variants: Both prompt variants have comparable debugging performance, but P4D-K produces more interpretable hard prompts because it retains the original prompt as part of the result.The framework’s optimization is described as a three-step procedure, with details referenced in Section 4.

5. Experiments

Experiments evaluate P4D on filtered concept- and object-related datasets against several safe T2I models, comparing prompt-search variants, safety-filter settings, transferability, and online platforms. P4D finds diverse problematic prompts, with performance affected by safety-filter constraints and substantial transfer across models.

  • Experimental setup: The evaluation filters prompts to those producing safe images with safe T2I models but unsafe images with standard T2I models.The datasets cover concept-related and object-related categories, including the I2P dataset.
  • Main results: P4D-N and P4D-K achieve comparable debugging results across safe T2I models and categories, while P4D-K preserves prompt interpretability.Their problematic prompts can also be unified into P4D-UNION, which increases failure rates because the discovered prompts are diverse.
  • Main results: ESD shows the highest P4D failure rate for nudity, despite initially providing the strongest protection against that category among the evaluated safe models.The authors attribute this vulnerability to concept removal learning associations for certain words without resisting optimized prompts.
  • Ablation and discussion: Disabling safety text filters during optimization enables P4D to identify more problematic prompts in SLD and SD-NEGP.The authors hypothesize that filters constrain the searchable textual embedding space, producing information obfuscation during debugging.
  • Ablation and discussion: The authors recommend aggregating problematic prompts found with different prompt lengths for more efficient red-teaming.This recommendation follows qualitative observations that optimized prompts may be shorter under some settings.
  • Transferability: Prompts found for ESD transfer to other safe models at rates above 60%, while over 70% of prompts found for other models bypass SLD-STRONG.Additionally, 37.28% of P4D-N prompts jailbreak all four safe T2I models.
  • Online platforms: P4D-K’s universal nudity prompts transfer to SDXL and Midjourney but are less effective against DALL·E 3.DALL·E 3 refuses some prompts and usually removes the target nudity concepts from outputs.

6. Conclusion

P4D is an automated red-teaming tool that finds problematic prompts exposing weaknesses in safety mechanisms for text-to-image diffusion models. Extensive experiments demonstrate its effectiveness for testing the reliability of safe T2I models.

  • P4D automatically finds problematic prompts that can produce inappropriate images bypassing deployed safety mechanisms.
  • P4D provides developers with a debugging and red-teaming tool for safeguarding and testing safe T2I models.

Impact Statement

The paper frames P4D as a red-teaming framework for advancing safety research in text-to-image models. It also identifies prompt-related issues that can be exploited to generate objectionable outputs, while the appendix specifies an implementation choice for guidance-based methods.

  • P4D automatically and efficiently identifies vulnerabilities in text-to-image models as a red-teaming framework for safety research.
  • Prompt dilution, information obfuscation, and semantic misalignment are discussed as issues exploitable for generating objectionable outputs.
  • For guidance-based methods SLD and SD-NEGP, the appendix deactivates the text filter during prompt optimization unless otherwise specified.

A.1. More Qualitative Results.

Additional qualitative and transferability experiments show that P4D-discovered prompts can affect multiple safe T2I models. The appendix also applies the identified vulnerabilities to formulate defenses against similar attacks.

  • More Qualitative Results: Additional P4D-N examples show problematic prompts alongside images generated by different safe T2I models.
  • More Qualitative Results: Problematic prompts found in ESD have the highest transferability, while prompts found in SLD-STRONG are most vulnerable across all four safe T2I models.
  • More Qualitative Results: Over 50% of prompts found by P4D can red-team multiple safe T2I models.
  • More Qualitative Results: Over 30% of problematic prompts found by both P4D-N and P4D-K are robust and general enough to red-team all safe T2I models simultaneously.
  • More Qualitative Results: The experiments use P4D-identified vulnerabilities to formulate defenses against similar attacks on SD-NEGP without adversarial training.

A.5. Random Seed Sensitivity Analysis

The random-seed analysis tests whether safe-model jailbreaking depends on the seed used during prompt optimization. It evaluates each problematic prompt over ten distinct sampled seeds and reports the resulting jailbreak counts and standard deviations.

  • Random Seed Sensitivity Analysis: Prompt optimization follows the I2P setting by using a fixed random seed for each prompt.
  • Random Seed Sensitivity Analysis: Table 11 reports random-seed sensitivity results using STD to denote standard deviation.
  • Random Seed Sensitivity Analysis: The sensitivity study samples 10 distinct seeds from [0, 2^31 −1] for each problematic prompt and counts seeds producing unsafe images.
  • Random Seed Sensitivity Analysis: The results report the average number of sampled seeds that jailbreak the safe T2I model for the same problematic prompt.

A.6. Text and Image Similarity Ablation Study

The ablation study examines whether optimized prompts preserve the semantics of their originals while bypassing safe T2I mechanisms. P4D-K retains interpretability while achieving similarity behavior comparable to P4D-N, whereas P4D-N trades textual similarity against text-image similarity as prompt length changes.

  • Similarity analysis: P4D produces high image similarity between images generated from original prompts with unconstrained T2I and optimized prompts with safe T2I.The method updates the optimized prompt based on image similarity during optimization.
  • P4D-K: Increasing K raises P4D-K’s text and text-image similarity, while reducing inserted tokens improves preservation of the input semantics.P4D-K remains similar to P4D-N while retaining interpretability.
  • P4D-N: Increasing N in P4D-N slightly improves text similarity but reduces text-image similarity, while image similarity remains much higher than text similarity.The study varies optimized prompt lengths and inserted token numbers for P4D-N and P4D-K.
  • Implementation: P4D uses an unconstrained and a safe T2I model, requiring approximately 20 to 30 minutes per prompt on two A5000 GPUs.The debugging duration varies with the safe T2I model.
  • Comparison with Advprompt: P4D consistently achieves a higher failure rate than the adapted Advprompt baseline across categories.The comparison attributes this advantage to P4D’s access to iterative diffusion decoding information.
  • Comparison with Advprompt: P4D optimizes one prompt in 20-30 minutes, compared with Advprompt’s 3-5 hours for the same task.Advprompt remains useful when model information is inaccessible because it is black-box.

A.11. Enhanced I2P dataset.

The authors release an enhanced I2P dataset containing universal problematic prompts discovered by P4D. It is intended to help T2I developers debug deployed safety mechanisms across different categories.

  • Dataset release: The enhanced I2P dataset collects universal problematic prompts found by P4D-N and P4D-K for debugging deployed T2I safety mechanisms.The released prompts cover different categories and are intended for model developers.
  • Dataset contents: The dataset includes prompts that are universal problematic prompts from Table 9, with both P4D-N and P4D-K variants provided.The dataset is released through Hugging Face.

B. Discussion and Limitations

The discussion evaluates P4D’s semantic fidelity and highlights limitations in prompt coherence and model access. Although optimized prompts can be unconventional, their generated images remain semantically close to the original outputs.

  • Prompt interpretability: P4D’s optimized prompts may lack linguistic coherence and contain sensitive words associated with the target concept.The authors argue that linguistic semantics are not required when unsafe generated images are the debugging objective.
  • Safety-mechanism behavior: Prompt dilution and incomplete removal of associated terms can help bypass safety mechanisms, especially for abstract targets such as nudity.The authors attribute the effectiveness of sensitive associated words to these two factors.
  • Semantic fidelity: High cosine similarity between original and optimized images indicates that unconventional P4D prompts can preserve the expected semantic domain.The similarity is used to select optimized prompts whose safe-model outputs resemble unconstrained-model outputs.
  • Access limitation: P4D requires access to both unconstrained and safe T2I models, while future work aims to reduce this white-box constraint.This requirement is motivated by the intended use of P4D as a debugging tool for model developers with model information.
Loading 2309.06135v3…