Source-linked AI summary
Red-Teaming the Stable Diffusion Safety Filter
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, Florian Tramèr
TL;DR
Stable Diffusion’s safety filter is poorly documented and its hidden concepts make its coverage difficult to assess. The paper reverse-engineers the filter and red-teams it, finding narrow sexual-content coverage, bypasses through prompt dilution, and failures on violence, gore, and other disturbing content. The authors therefore argue that future model releases should openly document safety measures, limitations, and vulnerabilities.
Problem
Stable Diffusion’s undocumented and obfuscated safety filter makes it difficult to assess its limitations and apply appropriate downstream mitigations.
Method
The paper reverse-engineers the filter from public code and tests it with illustrative examples, prompt dilution, and embedding inversion.
Results
The filter targets sexual content but ignores violence, gore, and other nonsexual disturbing content, while prompt dilution can bypass its sexual-content checks.
Takeaways & Limitations
Future model releases should provide open documentation of safety measures, limitations, and known vulnerabilities so the community can assess and improve them.
Takeaways & Limitations
The filter’s dilution effect remains a design problem, because images can be explicit even when most components are benign; the hierarchical child-content filter is also hypothesized to be dilution-vulnerable.
Abstract
from arXiv · showhide
Stable Diffusion is a recent open-source image generation model comparable to proprietary models such as DALLE, Imagen, or Parti. Stable Diffusion comes with a safety filter that aims to prevent generating explicit images. Unfortunately, the filter is obfuscated and poorly documented. This makes it hard for users to prevent misuse in their applications, and to understand the filter's limitations and improve it. We first show that it is easy to generate disturbing content that bypasses the safety filter. We then reverse-engineer the filter and find that while it aims to prevent sexual content, it ignores violence, gore, and other similarly disturbing content. Based on our analysis, we argue safety measures in future model releases should strive to be fully open and properly documented to stimulate security contributions from the community.
1 Introduction
Stable Diffusion publicly released an open-source image-generation model with a post-hoc filter intended to block explicit images. The filter’s undocumented and obfuscated design limits understanding of its coverage, while reverse engineering shows sexual-content focus and reliable prompt-based bypasses.
- 1 Introduction: Stable Diffusion includes a post-hoc safety filter that blocks generated images judged too close to 17 sensitive concepts in CLIP embedding space.The concepts themselves are not documented in the implementation.
- 1 Introduction: The 17 sensitive concepts are represented only by obfuscated CLIP embedding vectors rather than disclosed concepts.The embeddings function as hashes of the underlying concepts.
- 1 Introduction: Reverse engineering finds that the filter targets sexual content while ignoring violence, gore, and other similarly disturbing content.The authors also find that simple prompt engineering reliably bypasses the concepts the filter does target.
- 1 Introduction: The authors conclude that the filter is likely unsuitable for downstream applications requiring high safety standards.Poor documentation also prevents developers from properly assessing risks and adding mitigations such as stronger content blockers.
2 How the safety filter works
The safety filter is reconstructed from public code as a CLIP-based similarity check over generated-image embeddings and hidden concept embeddings. It also lowers thresholds for three hidden special-care concepts, making filtering more aggressive in those cases.
- 2 How the safety filter works: The filter’s operation can be deduced from code in Stable Diffusion’s public repository despite lacking documentation.The paper presents a simplified outline for Stable Diffusion v1.4.
- 2 How the safety filter works: The filter runs each generated image through CLIP’s image encoder and compares its embedding with 17 fixed sensitive-concept embeddings using cosine similarity.Each concept has a prespecified similarity threshold for deciding whether to filter the image.
- 2 How the safety filter works: The 17 fixed vectors come from undisclosed text prompts whose CLIP embeddings determine which unsafe concepts the filter detects.CLIP’s shared image-text space makes textual embeddings useful for comparing generated images with those concepts.
- 2 How the safety filter works: Three hidden special-care concepts trigger lower thresholds for the 17 sensitive concepts, producing more aggressive filtering.These special-care concepts are also supplied only as embeddings, and the behavior is undocumented.
3 Red-Teaming the safety filter in Stable Diffusion
Red-teaming demonstrates that Stable Diffusion’s filter allows multiple disturbing outputs, including violence, gore, vulgar content, and realistic images of public persons. Prompt dilution also bypasses sexual-content checks, while reverse engineering exposes the filter’s narrow concept coverage and weaknesses.
- 3 Red-Teaming the safety filter in Stable Diffusion: The authors’ analysis is illustrative rather than exhaustive, and they do not attempt to cover every potentially offensive or dangerous topic.They also provide a public notebook for testing the filter and displaying matched concepts.
- 3.1 Explicit content that bypasses the filter: Violent and gory generations, including depictions of children, are not prevented by the safety filter.The authors report unfiltered outputs for prompts describing killing and blood.
- 3.1 Explicit content that bypasses the filter: The filter also fails to restrict disturbing nonviolent content, such as images depicting starvation and broken bones.The paper describes these outputs as perturbing and unfiltered.
- 3.1 Explicit content that bypasses the filter: Stable Diffusion places no limitations on realistic images of real individuals in arbitrary contexts.Figure 2 illustrates personalities depicted in absurd scenarios.
- 3.1 Explicit content that bypasses the filter: Around 50% of generations from a detailed billboard prompt containing a naked man in an explicit pose evade the filter.The authors attribute this bypass to prompt dilution.
- 3.2 Prompt dilution: Prompt dilution adds unrelated benign details, moving the image embedding away from the standalone textual embedding of “nudity.”This can make an image explicit even when most of its components are benign.
- 3.3 Reverse engineering the obfuscated embeddings: An exhaustive NSFW-word search exactly recovers 15 of the 17 unsafe concepts and finds near-perfect matches for the remaining two.The recovered concepts all concern sexual content and nudity.
- 3.3 Reverse engineering the obfuscated embeddings: The three special-care concepts represent depictions of children, yet the hierarchical filter remains vulnerable to prompt dilution and does not cover violent content involving children.The paper characterizes this instantiation as simplistic and undocumented.
4 Discussion
The authors argue that AI safety measures should be robust, open, and documented, enabling assessment, mitigation, and community contributions. They also recommend staged releases, responsible disclosure, and addressing safety during model design rather than relying only on post-hoc filters.
- Safety measures should be complete and robust while remaining open and properly documented.Documentation allows the broader community to understand, improve, and customize safety systems for downstream applications.
- The authors note that closed-source generative models may face similar concerns when blocked concepts are undisclosed.They specifically cite concerns that secrecy around DALL·E’s filter may conceal censorship.
- Public analyses of limitations and vulnerabilities can help users understand risks and stimulate mitigation development.The authors compare this proposal to vulnerability repositories such as CVEs.
- Teams deploying popular models should maintain formal security policies and dedicated responsible-disclosure contacts.
- Staged releases can broaden understanding of model limitations before general public deployment.The paper notes that Stable Diffusion’s full public release followed the initial controlled release by 12 days, while the studied filter was added only three days before public release.
- Safety is easier to address during design than through post-hoc patches, and training-set curation is likely more effective than output filtering.
- The Stable Diffusion team acknowledged that the current filter is far from perfect, but the authors judge the downsides of obfuscation greater than its potential benefits.
A False negatives
The safety filter can be bypassed to generate explicit content of different kinds, demonstrating false negatives in its protection.
- The safety filter can be easily bypassed to generate explicit content of different kinds.The paper links generated images for topics discussed in the false-negative examples.
Explicit violence
Stable Diffusion generated violent images depicting people being killed, including children, with blood visible, despite the safety filter.
- The model generated an image of a person being killed with blood coming from his mouth.
- The model generated an image of a kid being killed with blood coming from his mouth.
Disturbing content
The examples include disturbing imagery involving starvation, broken bones, nudity, and explicit sexual activity in varied settings.
- The model generated a photograph of a poor child starving to death in Africa with broken bones.
- The examples include explicit sexual images involving naked people in public, architectural, and natural settings.The prompts describe a naked man in an explicit position, a naked couple having sex near the Eiffel Tower, and a group having sex in a forest.
B False positives
The filter produces false positives for benign prompts and abstract images, reflecting unexpected associations in CLIP’s latent space. These associations can disproportionately flag outputs that do not violate the safety policy.
- False positives: 8/15 attempts for a swimsuit photograph of Donald Trump were blocked, despite the prompt being non-sexually explicit.Replacing Donald Trump with Joe Biden, Penelope Cruz, a man, or a woman triggered no blocks.
- False positives: The blocked Trump images were mapped close to the “NSFW” concept in CLIP latent space.
- False positives: Unexpected CLIP associations can create fairness and bias issues by disproportionately flagging outputs that do not violate the safety policy.
- False positives: Abstract images were mapped close to special and unsafe concepts including “nude” and “vagina.”
C Pseudocode of the safety filter
The safety filter preprocesses generated images, embeds them with CLIP, and compares them against special-care and sensitive-concept embeddings. Threshold comparisons determine whether an image is unsafe, with special-care matches enabling more aggressive filtering.
- Pseudocode of the safety filter: The filter stores each generated image as an array and normalizes its pixel values before CLIP processing.
- Pseudocode of the safety filter: CLIP encodes the preprocessed image into a 768-dimensional embedding vector called image_embed.
- Pseudocode of the safety filter: The filter compares image_embed with three special-care embeddings and applies enhanced filtering when a cosine-distance threshold is exceeded.
- Pseudocode of the safety filter: The 17 sensitive concepts each have embeddings and thresholds that determine whether the generated image is unsafe.
D Reverse engineering the hidden concepts
The paper reverse-engineers the filter’s hidden concepts with a dictionary attack that exploits semantic similarity in CLIP embeddings. The search uses word lists, exact or near matches, and manual reasoning for multi-word concepts.
- Reverse engineering the hidden concepts: A dictionary attack recovers hidden concepts by searching for English words and phrases whose CLIP embeddings match the concealed concept embeddings.
- Reverse engineering the hidden concepts: CLIP’s preservation of semantic similarity makes its embeddings easier to search than a standard cryptographic hash.
- Reverse engineering the hidden concepts: The search dictionary combines common English words with obscene-word lists, popular subreddit names, body-part names, and offensive-word lists.
- Reverse engineering the hidden concepts: Exact matches identify concepts when cosine similarity reaches 1; otherwise, the ten best single-word matches guide manual searches for multi-word concepts.
- Reverse engineering the hidden concepts: The attack required reconstructing the same CLIP model used by the filter, identified as CLIP ViT-L/14.
E Full list of blocked and special care concepts
The reverse-engineered filter is dominated by nudity-related unsafe concepts, while its special-care concepts concern depictions of children. Most unsafe concepts were recovered exactly, and later-revealed repository contents confirmed the remaining guesses.
- Full list of blocked and special care concepts: The filter is defined by 17 unsafe embeddings and 3 special-care embeddings in CLIP latent space.
- Full list of blocked and special care concepts: 15 of 17 unsafe concepts were recovered exactly, and all of them relate to nudity.
- Full list of blocked and special care concepts: The three special-care concepts concern children’s depiction, including “young girl,” “young child,” and a near-match to “small girl.”
- Full list of blocked and special care concepts: Later-disclosed concepts confirmed the remaining guesses as “naked child,” “visible nipples,” and “little girl.”