Source-linked AI summary
Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models
Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, Yang Zhang
TL;DR
The paper investigates the risk that Text-to-Image models can be misused to generate unsafe images and targeted hateful memes. It constructs an unsafe-image typology, evaluates four models and prompt datasets, and tests image-editing methods for hateful-meme generation. The study finds substantial unsafe generation, including a 14.56% overall rate and 24% successful hateful-meme variants using DreamBooth.
Problem
The paper asks how prone Text-to-Image models are to deliberate unsafe generation and whether adversaries can automatically generate harmful memes targeting individuals or communities.
Method
The study defines five unsafe-image categories, assesses four models with harmful prompts, and evaluates DreamBooth, Textual Inversion, and SDEdit for hateful-meme variants.
Results
14.56% of images across four models and four prompt datasets are unsafe, and 24% of DreamBooth-generated memes successfully target specific individuals or communities.
Takeaways & Limitations
The findings highlight substantial risks from unsafe generation and support stronger safeguards across training-data curation, prompt regulation, and post-processing safety classification.
Takeaways & Limitations
The authors do not publicly disclose the keywords and phrases used, providing them only upon research request.
Abstract
from arXiv · showhide
State-of-the-art Text-to-Image models like Stable Diffusion and DALLE$\cdot$2 are revolutionizing how people generate visual content. At the same time, society has serious concerns about how adversaries can exploit such models to generate unsafe images. In this work, we focus on demystifying the generation of unsafe images and hateful memes from Text-to-Image models. We first construct a typology of unsafe images consisting of five categories (sexually explicit, violent, disturbing, hateful, and political). Then, we assess the proportion of unsafe images generated by four advanced Text-to-Image models using four prompt datasets. We find that these models can generate a substantial percentage of unsafe images; across four models and four prompt datasets, 14.56% of all generated images are unsafe. When comparing the four models, we find different risk levels, with Stable Diffusion being the most prone to generating unsafe content (18.92% of all generated images are unsafe). Given Stable Diffusion's tendency to generate more unsafe content, we evaluate its potential to generate hateful meme variants if exploited by an adversary to attack a specific individual or community. We employ three image editing methods, DreamBooth, Textual Inversion, and SDEdit, which are supported by Stable Diffusion. Our evaluation result shows that 24% of the generated images using DreamBooth are hateful meme variants that present the features of the original hateful meme and the target individual/community; these generated images are comparable to hateful meme variants collected from the real world. Overall, our results demonstrate that the danger of large-scale generation of unsafe images is imminent. We discuss several mitigating measures, such as curating training data, regulating prompts, and implementing safety filters, and encourage better safeguard tools to be developed to prevent unsafe generation.
1 Introduction
The paper examines how adversaries may misuse realistic Text-to-Image models to generate unsafe content, including hateful memes. It assesses model safety and evaluates image-editing methods for producing targeted hateful meme variants.
- Text-to-Image models can generate realistic images rapidly, raising concerns about their misuse for unsafe content and large-scale online dissemination.
- The study defines five unsafe-image categories: sexually explicit, violent, disturbing, hateful, and political.
- The authors assess four Text-to-Image models using harmful prompts from 4chan, Lexica, and a template-based dataset.
- The hateful-meme evaluation combines Stable Diffusion with DreamBooth, Textual Inversion, and SDEdit to generate variants targeting specific individuals or communities.
- 14.56% of images generated across four models and four prompt datasets are unsafe, while Stable Diffusion has an unsafe rate of 18.92%.
- The paper proposes curating training data, regulating prompts, and applying post-processing safety classifiers as mitigation measures.
2 Background
Text-to-Image models translate natural-language prompts into synthetic images, while image-editing methods learn image representations that support prompt-guided variants. The section distinguishes DreamBooth, Textual Inversion, and SDEdit by how they adapt or use the model.
- 2.1 Text-To-Image Models: Text-to-Image models use natural-language prompts to generate synthetic images through a language-understanding component and an image-generation component.
- 2.1 Text-To-Image Models: The study selects four pre-trained models based on popularity, public availability, and previously disclosed risks of unsafe image generation.
- 2.2 Image Editing Methods: Image editing learns a given image’s distribution into a method-specific representation, then generates variants guided by new prompts.
- 2.2 Image Editing Methods: DreamBooth fine-tunes the entire Text-to-Image model using several similar images and a prompt containing a special character.
- 2.2 Image Editing Methods: Textual Inversion optimizes a special-character embedding while keeping the model parameters frozen, whereas SDEdit iteratively denoises images using a diffusion model’s generative prior.
3 Preliminary Investigation
The preliminary investigation establishes a data-driven scope for unsafe images and characterizes generated unsafe content using harmful real-world prompts, clustering, and thematic coding. Five categories define when a synthesized image is considered unsafe.
- The investigation characterizes unsafe images generated by Text-to-Image models before the paper’s later analyses.
- Prompts are collected from 4chan and Lexica because both provide real-world text likely to elicit unsafe images.
- 4chan posts are filtered using syntactic structure analysis to improve prompt quality and reduce unnatural generations caused by noisy language.
- The study identifies 4,840 unsafe images, representing 39.90% of generated images, and clusters them into 16 groups using CLIP embeddings and K-means.
- Thematic coding of representative images yields five unsafe categories: sexually explicit, violent, disturbing, hateful, and political.
- A synthesized image is classified as unsafe when it presents at least one of the five categories.
4 RQ1: Safety Assessment
The study assesses unsafe-image generation across four Text-to-Image models and four prompt datasets using a multi-headed safety classifier, then examines factors associated with differing risks. Unsafe outputs are common, with risk varying by model, prompt design, training-data cleanliness, and prompt comprehension.
- Safety Classifier: The Multi-headed SC achieves 0.90 accuracy, 0.87 precision, 0.78 recall, and 0.82 F1-Score, outperforming the best fine-tuned Q16 baseline.
- Prompt-Level Analysis: 50.56% is the average unsafe-image probability for Template prompts, indicating that carefully designed prompts can elicit unsafe generation.
- Model-Level Analysis: 18.92% of Stable Diffusion’s generated images are unsafe, compared with 7.16% for DALL·E 2.
- Model-Level Analysis: Stable Diffusion and LDM generate more sexually explicit, disturbing, and political images than other models, while DALL·E mini generates 8.06% disturbing and 2.72% hateful images.
- 14.56% of generated images are unsafe on average across four Text-to-Image models and four prompt datasets.
- Potential Reasons: An estimated 3.46%-5.80% of training images are unsafe, while the relationship between unsafe training and generated images varies across categories.
- Potential Reasons: BLIP descriptiveness ranges from 0.37 to 0.40 for SD, LDM, and DALL·E mini, compared with 0.31 for DALL·E 2 on harmful prompts.
5 RQ2: Hateful Meme Generation
The paper examines whether Text-to-Image models can generate hateful meme variants, especially through image editing, and finds substantial success with Stable Diffusion. Generated variants preserve target-meme features, express target entities, and can be produced rapidly at scale.
- 5 RQ2: Hateful Meme Generation: Advanced image editing methods enable adversaries to generate hateful meme variants even when direct prompting with meme names fails.The evaluated methods are DreamBooth, Textual Inversion, and SDEdit applied to Stable Diffusion.
- 5.4 Results: Generated variants achieve image fidelity comparable to real-world variants, with SDEdit reaching a mean image fidelity of 0.78 versus 0.79 for original variants.Generated variants’ mean text alignment ranges from 0.16 to 0.18, below the original-variant mean of 0.22 but sufficient for target entities to appear evidently.
- 5.4 Results: SDEdit preserves the most target-meme features, while DreamBooth presents target entities most strongly, with mean text alignment of 0.18.DreamBooth and Textual Inversion primarily preserve facial features, whereas SDEdit excels at preserving posture.
- 5.4 Results: Image fidelity and text alignment trade off: increasing fidelity reduces alignment, preventing simultaneous optimization of both metrics.For DreamBooth and SDEdit, text alignment decreases as mean image fidelity rises.
- 5.4 Results: 24% of DreamBooth-generated variants are successful on average, compared with 10% for SDEdit and 9% for Textual Inversion.Across methods, the average success rate is 14%; Happy Merchant variants succeed more often than Pepe the Frog variants.
- 5.6 Real-World Impact Discussion: Text-to-Image models offer greater speed, scalability, and lower skill requirements than human creators for producing hateful meme variants.DreamBooth fine-tuning on Happy Merchant takes about 15 minutes on an NVIDIA DGX-A100, after which numerous variants can be generated.
6 Mitigating Measures
The paper discusses mitigation across the Text-to-Image supply chain, targeting unsafe content in training data, user prompts, and generated outputs.
- 6 Mitigating Measures: Mitigations include curating training data, regulating user-input prompts, and applying post-processing image safety classification.The authors estimate that 3.46%-5.80% of training-dataset images are unsafe and encourage more rigorous filtering before training.
7 Related Work
Prior work studied unsafe-image safety primarily through Stable Diffusion case studies and detection measures. This paper extends that work to automatically generated hateful memes and comparisons with real-world variants.
- 7 Related Work: Earlier studies mainly examined Stable Diffusion’s ability to generate unsafe image categories and the coverage of its built-in safety filter.The cited filter was reported to mostly detect sexual content.
- 7 Related Work: This paper investigates automatic generation of hateful memes and their variants using Text-to-Image models.It also compares generated variants with real-world hateful meme variants.
8 Discussion and Conclusion
The paper systematically assesses unsafe image and hateful meme generation from Text-to-Image models, finding substantial generation rates and discussing mitigation needs. Its hateful meme investigation shows that adversaries can automatically create realistic variants from a few hateful memes and image-editing methods.
- The paper presents the first systematic safety assessment of unsafe image and hateful meme generation from Text-to-Image models.
- Unsafe images can be generated at substantial rates when adversaries intentionally use harmful prompts, and some can also arise from harmless prompts.
- Up to 24% of generated meme variants share characteristics and features with real-world hateful meme variants.The paper states that these variants could be weaponized for hate campaigns on the Web.
- The findings motivate discussion of mitigating measures and the design of safer image-editing methods.
- An adversary can use a few hateful meme images and image-editing methods to generate realistic hateful memes automatically.
A.1 Ablation Studies
The ablation studies examine how guidance scale and prompt-designing strategies affect generated hateful meme variants, using image fidelity and text alignment as quality measures. Additional figures illustrate DreamBooth variants, target entities, and the effect of adding ChatGPT to prompt construction.
- Ablation Studies: The ablation studies investigate factors influencing the quality of generated hateful meme variants.
- Guidance Scale: Guidance scale controls the impact of the prompt on Stable Diffusion’s generated images, with larger values generally producing images that follow prompts more closely.
- Guidance Scale: DreamBooth variants are evaluated across guidance scales using image fidelity and text alignment, with Happy Merchant as the target meme.
- Prompt Designing: Prompt-designing strategies are compared by measuring the image fidelity and text alignment of generated variants.The evaluation appends the target entity to the end of a BLIP caption and separates them with a comma, while alternative strategies use only captions or only target entities.
- Generated Variants: The figures show original and generated Pepe the Frog variants produced with three image-editing methods using BLIP captions and target entities.
- Prompt Designing: Adding ChatGPT to the prompt-construction loop yields higher text-alignment values when image-fidelity values are held at the same level.