PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models

Patrick Schramowski, Manuel Brack, Björn Deiseroth, Kristian Kersting

arXiv:2211.05105v4cs.CVcs.AIcs.LG

TL;DR

Text-to-image diffusion models can inherit and reinforce inappropriate and biased behavior from unfiltered, imbalanced training data. The paper introduces I2P to evaluate this degeneration and SLD to suppress unsafe content during diffusion using the model’s own representations. SLD reduces inappropriate content without additional training, while strong dataset-learned representation biases are attenuated but not completely removed.

  • Problem

    Unfiltered and imbalanced training data can cause diffusion models to reproduce inappropriate and biased behavior, motivating systematic measurement and mitigation.

  • Method

    The paper introduces I2P and uses SLD, an inference-time extension of classifier-free guidance that steers diffusion away from inappropriate concepts without further tuning.

  • Results

    SLD removes and suppresses inappropriate image parts during diffusion with no additional training and no adverse effect on overall image quality.

  • Takeaways & Limitations

    SLD attenuates inappropriate generation and strong representation biases, supporting mitigation during generation while retaining the model’s learned concepts.

  • Takeaways & Limitations

    SLD does not completely remove strong representation biases, and I2P covers only specific concepts that may not reflect diverse opinions about inappropriateness.

Abstract

from arXiv · show

Text-conditioned image generation models have recently achieved astonishing results in image quality and text alignment and are consequently employed in a fast-growing number of applications. Since they are highly data-driven, relying on billion-sized datasets randomly scraped from the internet, they also suffer, as we demonstrate, from degenerated and biased human behavior. In turn, they may even reinforce such biases. To help combat these undesired side effects, we present safe latent diffusion (SLD). Specifically, to measure the inappropriate degeneration due to unfiltered and imbalanced training sets, we establish a novel image generation test bed-inappropriate image prompts (I2P)-containing dedicated, real-world image-to-text prompts covering concepts such as nudity and violence. As our exhaustive empirical evaluation demonstrates, the introduced SLD removes and suppresses inappropriate image parts during the diffusion process, with no additional training required and no adverse effect on overall image quality or text alignment.

1. Introduction

Text-conditioned diffusion models learn rich semantics from large multimodal datasets but can also reproduce inappropriate and biased behavior. The paper introduces I2P to measure this risk and SLD to suppress inappropriate content without further tuning.

  • Motivation: Large-scale, unfiltered training data can make diffusion models reproduce degenerated and biased human behavior.Such models may reflect biases in internet-scraped datasets and potentially replicate undesired behavior.
  • Motivation: Stable Diffusion generates inappropriate content, including for prompts that do not explicitly mention harmful material.The paper reports offensive, threatening, or anxiety-inducing outputs when analyzing SD trained on LAION-5B subsets.
  • Contributions: I2P is a benchmark of 4,703 real-world text-to-image prompts paired with inappropriateness scores from three detectors.It is designed to systematically evaluate inappropriate generations by diffusion models.
  • Contributions: SLD mitigates inappropriate degeneration by using the model’s learned knowledge of inappropriateness, requiring no external classifier or additional model tuning.The method manipulates the diffusion model’s latent space during generation.
  • Contributions: The paper presents SLD as an ethical image-editing approach for counteracting inappropriate degeneration in diffusion models.The authors describe it as, to their knowledge, the first work to consider image editing from this ethical perspective.

2. Risks and Promises of Unfiltered Data

Unfiltered and imbalanced datasets can transfer reporting and social biases into Stable Diffusion, while existing post-hoc safety measures may not reliably prevent inappropriate generation. The paper motivates dataset investigation alongside inference-time mitigation.

  • Risks: Large-scale models can reflect and reinforce biases present in unfiltered training data rather than counteracting them.The paper connects these risks to internet-scraped datasets and reporting bias.
  • Training data: Stable Diffusion is trained on over 2.32 billion English image-text pairs from LAION-2B-en and related subsets.The training data was not fully separated from annotated inappropriate content in LAION-5B.
  • Ethnic bias: 90% of the 100 nearest LAION images for “japanese body” showed explicit nudity, versus a 22% global average across 50 selected countries.The search was conducted in CLIP space and illustrates reporting bias in the dataset.
  • Ethnic bias: Over 75% of images generated for Japan using “<country> body” contained nudity, compared with a 35% global average.The study used NudeNet to detect exposed genitalia and found European, Asian, and Oceanic countries more associated with nudity than African or American ones.
  • Safety measures: Stable Diffusion’s post-hoc NSFW safety checker can be bypassed or deactivated on platforms offering image generation.The authors observed unrestricted generation of examples that the checker would otherwise withhold on lexica.art.
  • Contributions: The paper introduces SLD and the I2P benchmark as its two main contributions.These contributions address mitigation and systematic evaluation of inappropriate generation.

3. Safe Latent Diffusion (SLD)

SLD extends classifier-free guidance with an opposing safety concept during latent diffusion. Warm-up, thresholding, clipping, and momentum balance suppression of inappropriate content against preservation of image composition and fidelity.

  • Classifier-free guidance: Classifier-free guidance steers an unconditional noise estimate toward a text-conditioned estimate without requiring an external classifier.The guidance scale determines the magnitude of the prompt’s influence.
  • Safety guidance: SLD combines the original text prompt with an inappropriate concept and uses three noise predictions to guide generation away from that concept.The resulting prediction moves the unconditioned estimate toward the prompt-conditioned estimate while opposing the safety-conditioned estimate.
  • Safety guidance: Thresholded, element-wise safety guidance selectively scales dimensions that would move generation toward the inappropriate concept.The method clips the scaling factor to avoid image artifacts, while larger λ and sS produce stronger shifts.
  • Stability controls: A warm-up parameter delays safety guidance so higher values produce less significant changes and preserve overall image composition.The method also adds momentum to accelerate guidance over time for dimensions that are repeatedly directed toward unsafe content.
  • Process: During diffusion, safety guidance removes or suppresses inappropriate image parts as they are constructed in latent space.The paper visualizes this progression across diffusion steps and provides corresponding pseudo-code.

4. Configuring Safe Latent Diffusion

SLD offers configurable safety guidance that suppresses inappropriate concepts with adjustable editing strength. The recommended settings balance safety adaptation against preserving the original image, while stronger settings provide more aggressive suppression.

  • Safety concept: SLD defines an inappropriate safety concept through text, allowing concepts to be added or removed from the safety definition.The concept includes hate, violence, self-harm, sexual content, nudity, illegal activity, and related categories.
  • Hyper-parameters: The safety threshold λ determines the latent-space boundary between appropriate and inappropriate content, with recommended values λ ∈[0.0, 0.03].The threshold is theoretically restricted to [−1, 1], but the authors recommend conservative small positive values.
  • Hyper-parameters: The safety guidance scale sS controls suppression strength, with recommended values sS ∈[100, 3000] and no observed artifact effect from large sS.Larger values increase the number of latent values driven toward 1, unlike high image-guidance scales sg.
  • Hyper-parameters: The warm-up period δ delays strong safety guidance so initial image composition can form before SLD revises inappropriate regions.The authors recommend δ ∈[5, 20] and no more than half the total diffusion steps.
  • Configuration sets: Four configurations progressively increase editing aggressiveness, and the appropriate choice depends on the use case and individual preferences.Momentum is especially useful when inappropriate concepts occupy substantial image regions and require stronger editing.
  • Configuration sets: Weak settings remove superficial blood or obvious pornographic nudity, whereas strong and maximum settings are needed for severe injuries or more subtle artistic nudity.For nudity, medium settings may produce a bikini, while strong and maximum settings progressively add more clothing.
  • Trade-offs: Strong enough parameters can drive inappropriate content to zero, but Hyp-Max may diverge from the goal of keeping changes minimal.Sensitive applications may therefore require post-hoc interventions such as SD’s safety checker.

5. Inappropriate Image Prompts (I2P)

I2P is a benchmark of real-world prompts designed to measure inappropriate image generation across seven content categories. Its collection process and annotations support reproducible evaluation while retaining uncertainty about whether individual prompts produce inappropriate images.

  • Dataset: I2P contains over 4.5k real-world text prompts likely to produce inappropriate content across a broad range of categories beyond nudity.The dataset and evaluation pipeline are publicly available for research use.
  • Scope: The benchmark defines inappropriate content using seven categories: hate, harassment, violence, self-harm, sexual content, shocking images, and illegal activity.The authors note that inappropriateness varies across cultures and evolves over time.
  • Collection: The dataset collects up to 250 prompts for each of 26 keywords and phrases describing the seven categories.Prompts were drawn from real-world user-generated content associated with image generation.
  • Collection: Prompts were crawled from the top 250 Lexica images retrieved by CLIP similarity for each keyword, so they are not guaranteed to generate inappropriate content.The authors state that the probability of inappropriate generation is nevertheless high in evaluation.
  • Dataset: After deduplication, I2P contains 4703 unique prompts assigned to at least one of the seven categories.The benchmark also includes predicted inappropriate-image percentages and reproduction hyper-parameters.
  • Dataset analysis: Prompt toxicity shows only a weak correlation with the inappropriateness of generated images, and frequent words show no apparent connection to inappropriate-image generation.Prompts average 20 tokens.

6. Experimental Evaluation

The evaluation uses I2P to measure Stable Diffusion’s inappropriate degeneration and SLD’s mitigation. SLD substantially lowers inappropriate outputs and moderates geographic disparities, but does not eliminate learned racial bias and benefits from dataset filtering.

  • Experimental protocol: The evaluation generates ten images for every I2P prompt and classifies outputs using combined Q16 and NudeNet predictions.An image is classified as inappropriate when either classifier outputs the relevant label.
  • Stable Diffusion: Without safety measures, Stable Diffusion generates inappropriate content with category-dependent probabilities ranging from 34% to 52%.Across 25 prompts, at least one output batch is expected to contain 96% inappropriate images.
  • SLD mitigation: The strongest SLD configuration reduces inappropriate-generation probability by over 75%, leaving 9% of generated images classified as inappropriate.The authors caution that Q16 is conservative and may produce false positives for Hyp-Max.
  • SLD mitigation: SLD’s strongest configuration also reduces expected maximum inappropriateness, although it substantially increases variance and produces outliers.The table reports probabilities and bootstrap expected maxima over the I2P benchmark, with standard deviations shown as subscripts.
  • Conclusion: Overall, SLD largely mitigates Stable Diffusion’s inappropriate degeneration using the model’s learned representations.The evaluation also examines whether this approach applies to reporting biases in training data.
  • Bias mitigation: SLD yields 75% less explicit content across countries, reducing Japan’s nude-content rate to 12.0% versus a global percentage of 9.25%.The percentages of nude images become more evenly distributed between countries.
  • Bias mitigation: With minor changes, SLD does not entirely remove racial bias, leaving a medium statistically significant correlation between country-specific nude-generation rates with and without SLD.The reported correlation is Spearman r = 0.52, with the null rejected at p = 0.01.
  • Bias mitigation: Dataset filtering reduces biased representations, while combining filtering with SLD achieves better mitigation than SLD applied to Stable Diffusion v1.4 alone.The authors identify this combination as a promising direction for future work.

7. Discussion & Limitations

The paper frames SLD as an intervention that suppresses inappropriate content during diffusion while preserving image quality and text alignment. It also highlights ethical risks, including censorship, reversed misuse, limited coverage of social norms, and incomplete removal of learned biases.

  • Ethical implications: SLD suppresses inappropriate content during diffusion rather than removing potentially inappropriate training samples entirely.The authors argue that retaining such knowledge may support targeting related material at inference, though this could require substantial manual labor for balanced-dataset research.
  • Limitations: SLD attenuates strong representation biases but does not completely remove them, so dataset curation remains necessary.The conclusion advocates careful use of unfiltered, clearly imbalanced datasets.
  • Ethical implications: The authors warn that SLD could be repurposed for censorship or reversed to generate mainly inappropriate content, motivating transparent disclosure of suppressed content.
  • Limitations: The I2P benchmark covers specific inappropriate concepts and may not represent diverse opinions or social norms about inappropriateness.The model’s learned representation may also reflect the dispositions of groups represented in its training data.
  • Image fidelity & text alignment: User studies on DrawBench indicate that users slightly preferred SLD outputs, suggesting no observed sacrifice in image quality or text alignment.The study compared SLD images with Stable Diffusion reference images, while standard scores increased slightly with stronger hyper-parameters.

8. Conclusion

The paper shows that unfiltered and imbalanced training data can transfer inappropriate degeneration and representation biases into text-to-image models. It introduces I2P and SLD to measure and mitigate these issues, with suppression during diffusion and no additional training or adverse overall quality effect.

  • Inappropriate degeneration transfers from unfiltered and imbalanced training datasets into text-to-image models.
  • I2P is an image-generation test bed containing dedicated image-to-text prompts for inappropriate concepts such as nudity and violence.
  • SLD uses classifier-free guidance to remove and suppress corresponding inappropriate image parts during diffusion without additional training.
  • SLD has no adverse effect on overall image quality, while strong representation biases are attenuated but not completely removed.

Warning: Blurred inappropriate images and the associated

The paper provides further details on findings associated with its ethnic bias experiment. These findings are presented as supplementary discussion rather than a standalone methodological result.

  • The passage introduces additional details about findings from the ethnic bias experiment.
  • The discussion is explicitly related to findings from the ethnic bias experiment.
  • The passage serves as a transition into further analysis of the ethnic bias results.

A.1. CLIP Analysis on LAION-2B-en

The appendix describes how ethnic-bias and inappropriate-content analyses were conducted across LAION data, Stable Diffusion, and I2P-related evaluations. It also records data-availability and prompt-quality limitations, plus comparisons involving Stable Diffusion v2.

  • Ethnic Bias: For 50 countries, the study retrieved 100 closest LAION-2B-en images for each “<country> body” caption and measured nude-image percentages.
  • Ethnic Bias: Japan exceeded 90% explicit material in the LAION-2B-en ranking, compared with a 22% global average.
  • REALTOXICITYPROMPTS: REALTOXICITYPROMPTS were excluded from the main prompt set because they often produced unnatural images containing seemingly random letter combinations.Figure 7 illustrates this issue, while Table 3 reports classifier-based probabilities for the toxic-prompt subset.
  • Limitations: Approximately 8% of LAION-2B-en images were missing because their source URLs were unavailable at download time.
  • I2P: The I2P benchmark contains seven inappropriate categories collected using keywords from hate, harassment, violence, self-harm, sexual, shocking, and illegal-activity concepts.The prompts were collected from Lexica, and examples visualize four Stable Diffusion generations per category.
  • Stable Diffusion v2: Stable Diffusion v2 applied rigorous sexual- and nudity-related dataset filtering; combining filtering with SLD produced the highest mitigation.The passage states that SLD on Stable Diffusion v1.4 outperformed standalone filtering in Stable Diffusion v2.

F. I2P Results

The I2P results examine expected maximum inappropriateness across prompt categories and document the study procedures used to assess image fidelity and text alignment.

  • Expected maximum inappropriateness: The expected maximum inappropriateness differs substantially across I2P categories as the number of generated prompts increases.The evaluation plots expected maximum inappropriateness continuously from 10 to 200 generations for each category.
  • Expected maximum inappropriateness: At 200 prompts, sexual content reaches at most 50% expected maximum inappropriateness, while shocking content approaches 100%.The authors attribute much of this difference to Q16’s high false-positive rate, whose impact amplifies when taking the maximum over many prompts.
  • Metric limitation: Because high false-positive rates can make large-N expected maxima unsuitable, the main paper reports results at N = 25.This limitation concerns the metric’s interpretability when conservative classifier decisions are amplified across many prompts.
  • Qualitative evaluation: Qualitative comparisons show Stable Diffusion generations with and without SLD across the seven I2P categories.The comparisons include hate, harassment, violence, self-harm, sexual, shocking, and illegal activity prompts.
  • User studies: The user studies generated 10 images per model configuration and prompt, with users comparing SD reference images against corresponding SLD images.Each image pair received judgments from three distinct annotators, producing 30 decisions for each prompt.

G.2. Details on Results

The detailed evaluation describes the user study and the SLD algorithm, including its configurable safety concept, guidance, warmup, threshold, and momentum parameters.

  • User study: User-study results measure image fidelity and text alignment for each hyperparameter configuration on DrawBench.Figure 14 reports these results, with error bars representing standard deviation across 30 user decisions per prompt.
  • User study: Increasing guidance strength shifted perceived image fidelity from indecision toward preference for safety-guided images.The authors associate this trend with greater visualization of positive sentiments, while text alignment showed a considerably smaller similar trend.
  • SLD algorithm: SLD uses classifier-free-guidance principles to move the diffusion score toward the prompt and away from an inappropriate concept.It combines unconditional, prompt-conditioned, and concept-conditioned noise predictions during latent diffusion.
  • SLD algorithm: The safety concept can be adapted according to user preferences.The algorithm exposes guidance scales, warmup, threshold, and momentum parameters for this control.

I. SLD Ablation Studies

The ablation studies examine how SLD hyperparameters affect image changes over diffusion, while the I2P datasheet details the benchmark’s construction, labels, and scope.

  • Hyperparameter effects: Longer warmup periods produce less significant changes to image composition, whereas larger thresholds alter images more strongly.These effects are compared in Figure 16.
  • Hyperparameter effects: Momentum values above 0.5 may cause downstream image artifacts, and higher momentum produces stronger image changes.Figure 17 isolates momentum while fixing the guidance scales, warmup period, and threshold.
  • Diffusion-process behavior: Stronger hyperparameter configurations diverge from the original image earlier in the diffusion process and change it more substantially.The strongest setting never yields inappropriate images at any point in the illustrated process.
  • Benchmark construction: I2P was created to benchmark inappropriate degeneration in text-to-image models across seven concepts, filling a gap left by text-only toxicity prompts.The categories are hate, harassment, violence, self-harm, sexual content, shocking images, and illegal activity.
  • Benchmark construction: I2P contains 4703 unique prompts collected from real-world user-generated Stable Diffusion prompts using 26 keywords and phrases.Prompts are assigned categories and include metadata such as toxicity scores and a hard-prompt flag.
  • Dataset scope: The benchmark intentionally contains potentially inappropriate data, while prompt toxicity has only a weak relationship with generated-image inappropriateness.Only 1.5% of 4702 prompts were toxic, and low-toxicity prompts could still generate inappropriate images.
Loading 2211.05105v4…