PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models

Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, Yang Zhang

arXiv:2305.13873v2cs.CVcs.CRcs.CYcs.LGcs.SI

TL;DR

Text-to-Image models raise concerns about unsafe image generation, while research lacks a comprehensive, rigorous definition of unsafe images. This paper assesses unsafe outputs and generated hateful-meme variants, finding that 14.56% of images across four models and four prompt datasets were unsafe and 24% of generated memes successfully targeted individuals or communities.

  • Problem

    The research community lacks a comprehensive, rigorous definition of unsafe images, amid concerns about Text-to-Image models’ misuse for unsafe image generation.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    The paper develops a data-driven unsafe-image typology and systematically assesses model outputs, then evaluates hateful-meme variants generated with image-editing methods using multiple quality metrics.

    Source: §1 · pp. 2–3, §3.2 · p. 4

  • Results

    14.56% of generated images across four models and four prompt datasets were unsafe; Stable Diffusion led at 18.92%, while 24% of generated memes successfully targeted individuals or communities.

    Source: §1 · p. 2, §5.6 · p. 13

  • Takeaways & Limitations

    The findings underscore a need to strengthen safety measures in the image-generation process.

    Source: §1 · p. 3

    The prompt keywords and phrases are not publicly disclosed for ethical reasons and are available only upon research request.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

State-of-the-art Text-to-Image models like Stable Diffusion and DALLE$\cdot$2 are revolutionizing how people generate visual content. At the same time, society has serious concerns about how adversaries can exploit such models to generate unsafe images. In this work, we focus on demystifying the generation of unsafe images and hateful memes from Text-to-Image models. We first construct a typology of unsafe images consisting of five categories (sexually explicit, violent, disturbing, hateful, and political). Then, we assess the proportion of unsafe images generated by four advanced Text-to-Image models using four prompt datasets. We find that these models can generate a substantial percentage of unsafe images; across four models and four prompt datasets, 14.56% of all generated images are unsafe. When comparing the four models, we find different risk levels, with Stable Diffusion being the most prone to generating unsafe content (18.92% of all generated images are unsafe). Given Stable Diffusion's tendency to generate more unsafe content, we evaluate its potential to generate hateful meme variants if exploited by an adversary to attack a specific individual or community. We employ three image editing methods, DreamBooth, Textual Inversion, and SDEdit, which are supported by Stable Diffusion. Our evaluation result shows that 24% of the generated images using DreamBooth are hateful meme variants that present the features of the original hateful meme and the target individual/community; these generated images are comparable to hateful meme variants collected from the real world. Overall, our results demonstrate that the danger of large-scale generation of unsafe images is imminent. We discuss several mitigating measures, such as curating training data, regulating prompts, and implementing safety filters, and encourage better safeguard tools to be developed to prevent unsafe generation.

1 Introduction

This study assesses unsafe image generation across text-to-image models and examines their exploitation to create hateful meme variants. It finds substantial unsafe output and meme-generation risk, and discusses safeguards across the model supply chain.

Our Work

  • The safety assessment defines five unsafe-image categories and evaluates four models with harmful prompts from 4chan, Lexica, and a template dataset, plus MS COCO captions as a harmless baseline.The harmful prompts were chosen to measure a worst-case scenario in which an adversary deliberately seeks unsafe content.Source: §1 · p. 2

Main Findings

  • 14.56% of images across four models and four prompt datasets were unsafe; Stable Diffusion had the highest rate, at 18.92%.Template prompts elicited more unsafe content than the other datasets.Source: §1 · p. 2
  • 24% of meme variants generated with DreamBooth on Stable Diffusion combined features of the original hateful meme and the target individual or community.The variants had characteristics similar to a real-world hateful meme dataset; the study also examined DreamBooth, Textual Inversion, and SDEdit, and found ChatGPT could potentially increase meme diversity and quality.Source: §1 · p. 2
  • The image safety classifier exceeded Stable Diffusion’s built-in safety filter by 0.15 in accuracy, 0.28 in precision, 0.26 in recall, and 0.27 in F1-Score.Source: §1 · p. 2
  • The authors estimate that unsafe training images account for 3.46%-5.80% of training images, depending on the image model.Source: §1 · p. 2

Contributions

  • The paper recommends curating training data, regulating prompts, and applying post-generation safety classifiers to mitigate unsafe generation.The authors report using anonymous public datasets, author-only manual annotations, and aggregate reporting to reduce ethical risks.Source: §1 · p. 3

2 Background

Text-to-image models generate synthetic images from natural-language prompts, and image-editing methods use new prompts to create variants of existing images. The study examines four pretrained models and three editing methods.

2.1 Text-To-Image Models

  • Text-to-image models use a text encoder and image-generation component to turn prompts into synthetic images [51] [55].In Stable Diffusion [55], text-conditioned latent noise becomes an image embedding that the decoder converts into an image; examples include CLIP [50] or BERT [31] text encoders and diffusion [55] or VQGAN [69] generation components.Source: §2.1 · p. 3
  • The study selects Stable Diffusion [55], Latent Diffusion [55], DALL·E 2-demo [51], and DALL·E mini for popularity, public availability, and disclosed unsafe-generation risks [59].Stable Diffusion uses the sd-v1-4 checkpoint, pretrained on LAION-aesthetics v2 5+ (600 million image-text pairs) [61], while Latent Diffusion uses BERT rather than Stable Diffusion’s CLIP encoder and is pretrained on LAION-400M [63]. DALL·E 2-demo is a replica pretrained on a LAION-2B subset; DALL·E mini is used as an alternative to the unavailable official DALL·E model and is pretrained on Conceptual Captions (3M), Conceptual-12M, and YFCC-15M [29] [68].Source: §2.1 · p. 3

2.2 Image Editing Methods

  • Image editing learns a representation of a given image, then uses that representation and new prompts to generate variants [34] [44] [56].DreamBooth [56] fine-tunes the model using several similar images and a prompt with a special character; Textual Inversion [34] instead optimizes that character’s embedding while freezing model parameters. SDEdit [44] converts the image to a starting noise vector and denoises it conditioned on that vector and a new prompt, requiring neither model training nor special characters.Source: §2.2 · pp. 3–4

3 Preliminary Investigation

The preliminary investigation collects prompts likely to elicit unsafe images and uses generated images to develop a data-driven scope of unsafe content. It defines unsafe images through five categories and a clustering and coding analysis.

3.1 Prompt Collection for General Unsafe Image Generation

  • Researchers collected real-user unsafe prompts from 4chan and Lexica, yielding 2,470 toxicity-filtered 4chan sentences and 1,577 deduplicated Lexica prompts from 34 policy keywords.4chan sentences were selected for matching MS COCO caption syntax and Severe Toxicity scores above 0.8; Lexica was queried in November 2022.Source: §3.1 · p. 4

3.2 Scope of Unsafe Images

  • Q16 flagged 4,840 of 12,141 Stable Diffusion images (39.90%) as unsafe; CLIP embeddings and elbow analysis grouped them into 16 manually inspected clusters.The images came from 4,047 prompts, with three images generated per prompt and Stable Diffusion’s built-in safety filter removed.Source: §3.2 · pp. 4–5
  • Two-author coding of 160 centroid-nearest images yielded Fleiss’ kappa of 0.74 and five unsafe themes, alongside a safe miscellaneous theme.The authors developed and applied a codebook, then grouped similar codes into themes and assigned cluster themes by dominant code.Source: §3.2 · p. 5
  • Unsafe images are defined as showing at least one of five categories: sexually explicit, violent, disturbing, hateful, or political, aligned with DALL·E policy and Hive.The thematic analysis also identified a safe miscellaneous theme.Source: §3.2 · p. 5

4 RQ1: Safety Assessment

The assessment finds that all four text-to-image models can generate unsafe images, with risk varying by prompt dataset and model. It also examines classifier performance, unsafe training data, and models’ comprehension of harmful prompts.

4.1 Prompt Collection

  • The assessment used 500 descriptiveness-filtered 4chan prompts, 404 Lexica prompts, 30 Template prompts targeting five unsafe categories, and 500 MS COCO captions as a clean baseline.The 4chan prompts were selected using BLIP similarity; Lexica prompts came from category-related keyword searches, while Template prompts filled candidate phrases into a fixed prompt.Source: §4.1 · pp. 5–6

4.2 Image Generation

  • The study generated three images per prompt for Stable Diffusion, LDM, DALL·E 2, and DALL·E mini, totaling 6,000 4chan, 4,848 Lexica, 360 Template, and 6,000 MS COCO images.The keywords and phrases used in the prompts were withheld publicly for ethical reasons and offered only upon request for research purposes.Source: §4.2 · p. 6

4.3 Multi-Headed Safety Classifier

  • The Multi-headed SC achieved 0.90 accuracy, 0.87 precision, 0.78 recall, and 0.82 F1, compared with fine-tuned Q16’s 0.88, 0.77, 0.83, and 0.80, respectively.It also outperformed the baselines at reporting specific unsafe categories; the authors used it for the subsequent assessment.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 Safety Evaluation

  • 50.56% of images from carefully designed Template prompts were unsafe on average; harmless MS COCO prompts also elicited unsafe images from SD (0.27%), LDM (0.73%), DALL·E 2 (0.27%), and DALL·E mini (0.73%).The authors report that some images flagged from MS COCO prompts were indeed unsafe after manual inspection.Source: §4.4 · p. 7
  • SD and LDM generated sexually explicit (4.65%-5.85%), disturbing (4.79%-5.62%), and political (3.21%-5.83%) images more often than other models; DALL·E mini led in disturbing (8.06%) and hateful (2.72%) images.Figure 3 compares the proportions across five unsafe categories.Source: Figure 3 · p. 7, §4.4 · p. 7
  • 14.56% of generated images were unsafe on average across four models and prompt datasets; Stable Diffusion had the highest rate (18.92%), while DALL·E 2 had the lowest (7.16%).The four models showed different risk levels, with DALL·E mini and LDM between those extremes.Source: §4.4 · p. 7, Table 3 · p. 8
  • 3.46%-5.80% of sampled training images were estimated unsafe, but their prevalence was not necessarily significantly correlated with unsafe generation (Kendall’s tau: -0.33 overall).DALL·E mini had the lowest share of hateful training images (0.06%) yet generated the most hateful images among the models.Source: Table 4 · p. 8, §4.4 · p. 8
  • BLIP descriptiveness for harmful prompts was 0.37 to 0.40 for SD, LDM, and DALL·E mini, versus 0.31 for DALL·E 2, consistent with the former models’ higher risks.The authors use BLIP similarity between harmful prompts and their generated images as a measure of prompt comprehension.Source: Figure 4 · p. 8, §4.4 · p. 8

5 RQ2: Hateful Meme Generation

Stable Diffusion image-editing methods can generate hateful meme variants that retain meme features and depict target entities; DreamBooth achieves the highest manual success rate, and the process reduces the effort needed to produce variants at scale.

5.1 Preliminary Investigation

  • Meme-name prompts yielded four visually similar memes from DALL·E mini, including Pepe the Frog, while Stable Diffusion, LDM, and DALL·E 2 generated none of the 20 tested memes.The preliminary study generated 240 images and found almost perfect annotator agreement (Fleiss’ kappa 0.82); most models could not generate most memes from their names, motivating image-editing experiments.Source: §5.1 · p. 9

5.2 Threat Model

  • The adversary’s goal is to preserve a target meme’s features while making a target individual or community visually evident, assuming full access to the Text-to-Image model.A hateful meme variant combines the original meme’s features with characteristics of a target entity; the adversary may modify model parameters to personalize generation.Source: §5.2 · p. 9

5.3 Evaluation Process

  • The evaluation used Stable Diffusion, three image-editing methods, and prompts derived from real-world variants to compare image fidelity, text alignment, and manual success.The dataset contained 150 variants of Happy Merchant and Pepe the Frog; each prompt and target meme produced eight variants per method, for 2,400 variants per method. BLIP captions were combined with target entities, and manual assessment required retaining meme features and evidently presenting the entity.Source: §5.3 · pp. 9–11

5.4 Results

  • Generated variants were comparable to real-world variants: original variants averaged 0.79 image fidelity, while SDEdit reached 0.78; generated text alignment ranged from 0.16–0.18 versus 0.22 for originals.SDEdit had the highest mean image fidelity (0.78), while DreamBooth had the highest mean text alignment (0.18); the results show a trade-off between fidelity and alignment.Source: §5.4 · p. 11, Figure 7 · p. 12
  • Generated examples generally retained Happy Merchant’s features, while DreamBooth more clearly depicted target entities and SDEdit better preserved posture.DreamBooth and Textual Inversion often retained facial features such as the giant nose and beard; examples also showed generated entities including Hillary, UK, and Mexican.Source: §5.4 · p. 11, Figure 8 · p. 12
  • 14% of generated variants succeeded on average; DreamBooth reached 24%, compared with SDEdit at 10% and Textual Inversion at 9%.Success averaged 18% for Happy Merchant variants and 11% for Pepe the Frog variants; Table 5 reports the manually annotated success rates.Source: Table 5 · p. 12, §5.4 · p. 12

5.5 ChatGPT in the Loop

  • Rephrasing prompts with ChatGPT increased text alignment and produced examples that better depicted the target entity Facebook.The study generated 30 rephrased prompts and eight variants per prompt with DreamBooth, comparing them with the same number of variants using the unrephrased prompt.Source: §5.5 · pp. 12–13

5.6 Real-World Impact Discussion

  • DreamBooth fine-tuning took about 15 minutes on an NVIDIA DGX-A100, after which 24% of generated memes succeeded; prompt-based generation lowers skill requirements and enables faster, scalable production.The authors warn that selected high-quality generated memes could spread widely and support hate campaigns targeting individuals or communities.Source: §5.6 · p. 13

6 Mitigating Measures

The paper recommends mitigating unsafe generation throughout the Text-to-Image supply chain by filtering training data, regulating prompts, and applying image safety classification. Prompt filtering lowers unsafe-image generation, but open-source models and hateful memes present limitations for these defenses.

Regulating Prompts

  • 14.56% to 9.34%: filtering prompts containing 66 unsafe keywords reduced the percentage of unsafe images.The authors note this approach applies to online services, not open-source models such as Stable Diffusion.Source: §6 · p. 13

Image Safety Classification

  • The safety classifier achieved 90% accuracy across five categories, but flagged only 44.19% of successfully generated hateful memes as unsafe.It outperformed Q16 and Stable Diffusion’s built-in safety filter; the authors suggest including hateful memes in classifier training data.Source: §6 · p. 13

Curating Training Data

  • An estimated 3.46%-5.80% of training-dataset images are unsafe, motivating more rigorous data filtering before model training.The paper notes that datasets such as LAION-400M have already undergone some curation, including removal of NSFW images.Source: §6 · p. 13

7 Related Work

Prior work examined Stable Diffusion safety and real-world hateful meme variants, while this work extends safety assessment across models and prompt sources and investigates AI-generated meme variants. It also compares generated variants with real-world examples.

Safety of Text-To-Image Models

  • Prior studies examined Stable Diffusion, with case studies showing sexual, violent, and disturbing outputs and an I2P-based assessment of unsafe generation [52] [59].Source: §7 · p. 14
  • Existing safety classifiers mainly detect sexual content [52] or general inappropriate concepts, leaving unsafe-image categories and cross-dataset and cross-model risks unclear.Source: §7 · p. 14
  • This work uses a multi-headed classifier to predict unsafe-image categories and assesses Stable Diffusion, Latent Diffusion, DALL·E 2, and DALL·E mini across multiple prompt sources.Source: §7 · p. 14

Hateful Memes & Variants

  • Prior meme research examined variant popularity across web communities [70], hateful-meme detection [45] [49] [57] [72], and multimodal detection following the Hateful Meme Challenge.Source: §7 · p. 14
  • Unlike studies of manually created, real-world variants—including Happy Merchant and Pepe the Frog —this work investigates AI generation and compares generated variants with real-world ones.Source: §7 · p. 14

8 Discussion and Conclusion

Text-to-Image models can generate unsafe images even from harmless prompts, and adversaries can use image editing to produce hateful meme variants resembling real-world examples. The authors call for clearer definitions of unsafe content and stronger safeguards, while noting limitations in their assessment.

Discussion and Conclusion

  • Up to 24% of generated meme variants shared characteristics and features with real-world hateful meme variants, while unsafe images were also possible from harmless prompts.The authors describe the hateful meme variants as potentially weaponizable for online hate campaigns.Source: §8 · p. 14

Definition of Unsafe Content

  • The authors argue that researchers and AI practitioners should define unsafe content comprehensively to support holistic model audits, effective safeguards, and safer Text-to-Image models.They note that unsafe content is broad and currently lacks an accurate, comprehensive definition.Source: §8 · p. 14

Generation of Unsafe Content

  • Because models may generate unsafe content even from safe descriptions, the authors call for user awareness, safeguards against exploitation, and further study of Image-to-Text risks and societal effects.They highlight possible exposure of young people to unsafe content and potential mental-health consequences.Source: §8 · pp. 14–15

AI-Generated Hateful Memes

  • Adversaries can use a few hateful meme images and image-editing methods to generate realistic memes, motivating safeguards against fine-tuning on hateful subjects and tools to detect AI-generated images.The authors say detection could support online content moderation, including mitigation of orchestrated hate campaigns.Source: §8 · p. 15

Limitations

  • The assessment may omit unsafe concepts such as self-harm and may be affected by manual-annotation bias; crowdsourcing and user studies were avoided for ethical and expertise concerns.The authors acknowledge that their five-category scope may limit comprehensiveness and hope the work raises awareness of the need for accurate safeguards.Source: §8 · p. 15

A Appendix · A.1 Ablation Studies

The ablations examine how guidance scale and prompt design affect the fidelity and text alignment of DreamBooth hateful-meme variants. Lower guidance can reduce text alignment at matched fidelity, while combining a BLIP caption with a target entity generally improves alignment.

A.1 Ablation Studies

  • The ablation studies investigate factors that influence the quality of generated hateful-meme variants.Source: §A.1 · p. 17
  • When guidance scale reaches 7 or higher, text-alignment gaps between scales diminish.The authors use guidance scale 7 for the Section 5.3 evaluation.Source: §A.1 · p. 17
  • Entity-only prompts yield the lowest editability, reflected in low text alignment, because each entity is a single, insufficiently descriptive word.The comparison holds image fidelity at the same level.Source: §A.1 · p. 17
  • Prompt design plays an important role in the text alignment of generated variants.The ablation compares BLIP-caption-only, entity-only, and combined prompts.Source: §A.1 · p. 17
  • At matched image fidelity, variants generated with guidance scale 3 have lower text alignment than those using larger scales.The study evaluates DreamBooth variants of Happy Merchant.Source: Figure 10 · p. 17, §A.1 · p. 17
  • At matched image fidelity, BLIP caption + entity prompts produce the highest text alignment in most cases.The comparison tests BLIP caption alone, entity alone, and BLIP caption + entity.Source: §A.1 · p. 17, Figure 11 · p. 18

Related papers

Loading 2305.13873v2…