PaperlayerView exact revision on arXiv

Source-linked AI summary

Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?

Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, Chun-Ying Huang

arXiv:2310.10012v4cs.LG

TL;DR

T2I safety mechanisms’ effectiveness across diverse prompts remains underexplored despite concerns that diffusion models can generate copyrighted, prohibited, and NSFW content. Ring-A-Bell extracts sensitive concepts and automatically constructs problematic prompts without prior target-model knowledge, and experiments show these prompts expose substantial weaknesses in online services and concept-removal methods.

  • Problem

    The effectiveness of T2I safety mechanisms across a wide range of prompts remains largely unexplored despite risks of generating copyrighted, prohibited, and NSFW content.

  • Method

    Ring-A-Bell extracts holistic representations of sensitive concepts and uses them to identify problematic prompts without prior knowledge of the target model.

  • Results

    Ring-A-Bell reveals defects in online services and concept-removal methods, with problematic prompts increasing most concept-removal methods’ inappropriate-image success rates by more than 30%.

  • Takeaways & Limitations

    Ring-A-Bell provides an offline, model-agnostic red-teaming tool for assessing whether T2I models detect or remove inappropriate content.

  • Takeaways & Limitations

    The model-specific evaluation requires prior knowledge of the safety-equipped model and the unconstrained model, multiple samples, and similar model architectures.

Abstract

from arXiv · show

Diffusion models for text-to-image (T2I) synthesis, such as Stable Diffusion (SD), have recently demonstrated exceptional capabilities for generating high-quality content. However, this progress has raised several concerns of potential misuse, particularly in creating copyrighted, prohibited, and restricted content, or NSFW (not safe for work) images. While efforts have been made to mitigate such problems, either by implementing a safety filter at the evaluation stage or by fine-tuning models to eliminate undesirable concepts or styles, the effectiveness of these safety measures in dealing with a wide range of prompts remains largely unexplored. In this work, we aim to investigate these safety mechanisms by proposing one novel concept retrieval algorithm for evaluation. We introduce Ring-A-Bell, a model-agnostic red-teaming tool for T2I diffusion models, where the whole evaluation can be prepared in advance without prior knowledge of the target model. Specifically, Ring-A-Bell first performs concept extraction to obtain holistic representations for sensitive and inappropriate concepts. Subsequently, by leveraging the extracted concept, Ring-A-Bell automatically identifies problematic prompts for diffusion models with the corresponding generation of inappropriate content, allowing the user to assess the reliability of deployed safety mechanisms. Finally, we empirically validate our method by testing online services such as Midjourney and various methods of concept removal. Our results show that Ring-A-Bell, by manipulating safe prompting benchmarks, can transform prompts that were originally regarded as safe to evade existing safety mechanisms, thus revealing the defects of the so-called safety mechanisms which could practically lead to the generation of harmful contents. Our codes are available at https://github.com/chiayi-hsu/Ring-A-Bell.

1 INTRODUCTION

T2I diffusion models can generate inappropriate content because Internet-scale training data includes restricted images, motivating safety mechanisms and systematic red-team evaluation. Ring-A-Bell uses prompt engineering and concept extraction to test these mechanisms offline and model-independently, with experiments showing problematic prompts can substantially increase inappropriate-image generation.

  • Internet-derived training data enables broad T2I generation but can include restricted images, leading models to memorize and generate copyright-violating, prohibited, and NSFW content.
  • Existing safety mechanisms either constrain text embeddings during inference or fine-tune diffusion models to avoid copyrighted and inappropriate images.
  • Ring-A-Bell generates problematic prompts for red-teaming T2I safety mechanisms and revealing sensitive concepts such as nudity and violence.
  • Concept extraction uses CLIP or general text encoders, enabling model-independent prompt evaluation and efficient offline testing.
  • More than 30%: Ring-A-Bell can increase the success rate of most concept removal methods in generating inappropriate images.

2 RELATED WORK

Related work covers red-teaming tools for AI systems, prompt-engineering strategies, and safety mechanisms for T2I diffusion models. These approaches include detection-based filtering and removal-based steering away from sensitive content.

  • Red-teaming originated in cybersecurity and has expanded to machine-learning systems, including language models and more recent T2I models.
  • Prompt engineering modifies input text with carefully crafted prompts and includes hard discrete and soft continuous prompt approaches.
  • T2I safety methods are broadly divided into detection-based filtering and removal-based steering away from inappropriate content.

3 MAIN APPROACH

Ring-A-Bell develops model-specific and model-agnostic procedures for red-teaming safety mechanisms in text-to-image diffusion models. Its model-agnostic pipeline extracts concept representations from prompt pairs and uses them to generate problematic prompts offline without prior target-model knowledge.

  • Model-specific evaluation: The paper first formulates a model-specific attack by finding an adversarial concept whose constrained-model generation resembles the unconstrained model for inappropriate trajectories.The objective minimizes a loss based on the similarity of backward-process distributions under the original and adversarial concepts.
  • Model-specific evaluation: The model-specific approach requires target-model and unconstrained-model access, multiple samples for expectation estimation, and similar architectures for meaningful noise alignment.These requirements limit practical use when developing red-teaming tools with limited computational resources.
  • Model-agnostic evaluation: Ring-A-Bell replaces white-box attack construction with a black-box, model-agnostic pipeline that assumes access only to a general text encoder.Problematic prompts can be prepared offline and later deployed against online services or concept-removal methods.
  • Model-agnostic evaluation: Ring-A-Bell injects the extracted concept into a target prompt with a tunable strength coefficient and optimizes the resulting representation to produce a problematic prompt.The transformed prompt is designed to test whether a supposedly removed concept can be revoked through prompt optimization.
  • Model-agnostic evaluation: It extracts a holistic concept representation by subtracting embeddings of semantically similar prompt pairs that differ in the target concept, then averaging across pairs.The procedure is intended to reduce context-dependent influence while covering plausible semantic scenarios.

4 EXPERIMENTS

Ring-A-Bell is evaluated against online services, text-to-video generation, and concept-removal methods using problematic-prompt datasets and Attack Success Rate. Experiments show that prompt length, concept weighting, optimizer choice, and prompt-pair diversity affect attack performance.

  • Evaluation Setup: The evaluation uses 95 nudity prompts and 250 violence prompts from the I2P dataset, with NudeNet and Q16 classifying generated images.Each prompt produces one image with a fixed random seed.
  • Online Services: Ring-A-Bell is tested against Midjourney, DALL·E 2, Gen-2, and stability.ai using prompts generated for red-teaming concept-removal models.The study also evaluates Gen-2 as a text-to-video model for nudity and violence.
  • Online Services: Online services differ in protection: Ring-A-Bell bypasses safety checkers to generate nude images, while most services detect violence or bloodshed less effectively than nudity.DALL·E 2 is identified as the exception for violence detection, and other services generate images once prompts pass their checkers.
  • Concept Removal Methods: Ring-A-Bell more effectively recalls nudity and violence concepts than original prompts and QF-Attack, especially under the union setting when safety checkers are deployed.Safety checkers filter more explicit nudity images than violence images.
  • Ablation Studies: At η = 3, nudity attacks perform better; violence improves as η increases, but gains gradually diminish at sufficiently large values, including η = 5 and η = 5.5.The experiments use K = 38 for nudity and K = 77 for violence in this analysis.
  • Ablation Studies: GA and PeZ are competitive, but GA performs better on challenging ESD and SLD nudity settings and surpasses PeZ for violence.The comparison uses K = 16, η = 3 for nudity and K = 77, η = 5.5 for violence.

5 CONCLUSION

The paper demonstrates risks in online services and concept-removal methods for detecting or removing nudity and violence. Ring-A-Bell-generated problematic prompts can manipulate these models into generating inappropriate images.

  • Ring-A-Bell reveals underlying risks in online services and concept-removal methods targeting nudity and violence.
  • Problematic prompts generated by Ring-A-Bell can manipulate T2I models into successfully generating inappropriate images.
  • Ring-A-Bell serves as a red-teaming tool for assessing T2I models’ removal or detection of inappropriate concepts.

A DERIVATION OF MODEL-SPECIFIC EVALUATION

This section connects sequence-level KL divergence between original and modified diffusion distributions to an alternate loss. It also situates the approach within red-teaming, prompt engineering, and T2I safety mechanisms.

  • A DERIVATION OF MODEL-SPECIFIC EVALUATION: The derivation minimizes KL divergence between original and modified diffusion distributions to obtain an alternate loss.
  • A DERIVATION OF MODEL-SPECIFIC EVALUATION: KL divergence between the relevant normal distributions simplifies to a squared difference between their means when variance terms are ignored.
  • Red-Teaming Tools for AI: Red-teaming searches for vulnerabilities and weaknesses so organizations can strengthen defenses and cybersecurity protections.
  • Red-Teaming Tools for AI: Prior T2I red-teaming includes query-free prompt perturbations and online-forum prompts used to examine safety mechanisms.
  • Diverse Approaches in Prompt Engineering: Prompt engineering modifies input text and offers a computationally lighter alternative to fine-tuning large models.
  • Text-to-Image Diffusion Model with Safety Mechanisms: T2I safety mechanisms comprise detection-based methods and removal-based methods.
  • Text-to-Image Diffusion Model with Safety Mechanisms: Detection-based methods use safety checkers to assess generated images for correlations with sensitive or harmful concepts.
  • Text-to-Image Diffusion Model with Safety Mechanisms: Removal-based methods constrain generation or modify model parameters to eliminate sensitive concepts during image synthesis.

C LIMITATION

The paper illustrates inappropriate generations from online services and concept-removal models using Ring-A-Bell prompts. These examples cover nudity and violence, with comparisons between original and manipulated prompts.

  • C LIMITATION: Ring-A-Bell prompts generate inappropriate images from online services, as illustrated through additional examples in Figure 5.
  • C LIMITATION: The examples use blurred or otherwise publication-modified visualizations.
  • C LIMITATION: Concept-removal models generate additional nudity examples when tested with Ring-A-Bell prompts configured with K = 77 and η = 3.5.
  • C LIMITATION: Figure 7 compares violence images generated by concept-removal models from original prompts and Ring-A-Bell-generated prompts.

F VISUALIZATION OF CONCEPT RETRIEVAL IN CONCEPT REMOVAL METHODS

The paper visualizes Ring-A-Bell retrieval of removed concepts including cars and Van Gogh style. Manipulated prompts can restore concepts that removal methods suppress under original prompts.

  • F VISUALIZATION OF CONCEPT RETRIEVAL IN CONCEPT REMOVAL METHODS: Ring-A-Bell retrieves forbidden concepts such as cars and Van Gogh style across concept-removal methods.
  • F.1 CONCEPT OF CAR: ESD removes cars from images generated with original prompts, but Ring-A-Bell-generated prompts lead ESD to produce images containing cars.
  • F.1 CONCEPT OF CAR: Figure 8 visualizes SD and ESD outputs for original prompts alongside ESD outputs using Ring-A-Bell prompts.
  • F.2 CONCEPT OF VAN GOGH: Figure 9 compares SD, ESD, CA, and FMN on original prompts with manipulated-prompt outputs from ESD, CA, and FMN.
  • F.2 CONCEPT OF VAN GOGH: Each Figure 9 line represents one prompt, while Ring-A-Bell manipulates the original prompt to generate new prompts.
  • Concept Extraction: The method constructs empirical concepts from prompt pairs, using ChatGPT to create concept-containing and concept-excluding sentences.
  • Concept Extraction: For Van Gogh style, prompts add or remove the phrase “with Van Gogh style”; for nudity, concept-related vocabulary is permuted into scenarios.

I ABLATION STUDY OF THREE ATTACK STRATEGIES

The ablation study evaluates Ring-A-Bell alongside modification and dilution, finding that problematic images emerge only when all three strategies are combined. Modification avoids inappropriate words, while dilution prevents generated images from being identified as inappropriate.

  • Combining Ring-A-Bell with modification and dilution produces problematic images, whereas the latter two techniques applied to the original prompt do not generate nudity.The results indicate that Ring-A-Bell-generated prompts are necessary for the combined attack to produce inappropriate content.
  • Modification avoids inappropriate words in the problematic prompt through input filtering.
  • Dilution prevents generated images from being identified as inappropriate through output filtering.

J ADDITIONAL EXPERIMENTS ON DIFFERENT VALUES OF K

Additional experiments test smaller K values for the nudity concept and find that K = 8 performs worse than K = 16. The study uses K = 8 and η = 3 for the smaller-K setting.

  • K = 8 with η = 3 is evaluated for the nudity concept using the settings of Table 3.
  • Results obtained with K = 8 are not superior to those achieved with K = 16.
  • The experiments infer that small values of K are not conducive to effectively causing the model to generate inappropriate content.

K COMPARISON AGAINST P4D

The comparison against P4D distinguishes their evaluation metrics and reports that P4D-N degrades under Ring-A-Bell’s metric, while Ring-A-Bell outperforms P4D under P4D’s metric in most settings. The authors also note that their metric is more restrictive for identifying nudity-related content.

  • P4D’s ASR uses the NudeNet binary safe/unsafe classifier with an unsafe threshold of 0.45, whereas Ring-A-Bell uses the NudeNet detector for specific exposure parts.
  • Ring-A-Bell’s evaluation is more restrictive because it recognizes only certain parts as nudity-related, unlike images judged unsafe under the more lenient alternative.
  • P4D-N’s performance degrades heavily under Ring-A-Bell’s metric and does not stand out significantly compared with Ring-A-Bell.
  • Ring-A-Bell performs better than P4D under P4D’s metric in 3 out of 5 settings.

M DISCUSSION AND RESULT WITH SIMILAR WORKS

The discussion compares Ring-A-Bell with concurrent red-teaming approaches, emphasizing differences in feedback assumptions and performance on concept-removal methods. Ring-A-Bell is designed to identify problematic prompts where safety-filter feedback may not reveal failures.

  • Comparison with similar works: Mehrabi et al. use a language-model feedback loop that adjusts adversarial prompts from T2I outputs and safeness scores.
  • Comparison with similar works: Online services may directly reject inappropriate images, so Mehrabi et al.’s language model may receive no meaningful feedback in that setting.
  • Comparison with similar works: Ring-A-Bell incorporates prompt dilution when evaluating online T2I services to increase overall attack success rate.
  • Comparison with similar works: SneakyPrompt uses reinforcement learning, CLIP score rewards, a 0.26 early-stopping threshold, and an upper limit of 60 queries.
  • Comparison with similar works: SneakyPrompt performs much lower than Ring-A-Bell on concept-removal methods because safety-filter feedback can mark prompts safe after sensitive concepts have been removed.
Loading 2310.10012v4…