Source-linked AI summary
DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, Duen Horng Chau
TL;DR
Writing prompts for desired image details requires trial and error, and users often lack principled understanding of why prompts work. DiffusionDB addresses this by collecting a large-scale dataset of Stable Diffusion images, prompts, and hyperparameters, revealing prompt patterns and model errors while broadening access to research data.
Problem
Generating images with desired details requires proper prompts, but developing them can be difficult, random, and unprincipled.
Method
DiffusionDB collects Stable Diffusion images shared on a public Discord server with their prompts and hyperparameters, then analyzes prompt syntactic and semantic features.
Results
The dataset analysis reveals popular prompt phrases and image styles, and shows that very small diffusion steps or extreme image dimensions can produce blurry or dissimilar images.
Takeaways & Limitations
The released dataset, code, and CC0 license broaden public access to prompt-image data for studying prompt patterns and generative-model behavior.
Takeaways & Limitations
Findings from DiffusionDB may not generalize to other text-to-image models because different models can require different prompting patterns.
Abstract
from arXiv · showhide
With recent advancements in diffusion models, users can generate high-quality images by writing text prompts in natural language. However, generating images with desired details requires proper prompts, and it is often unclear how a model reacts to different prompts or what the best prompts are. To help researchers tackle these critical challenges, we introduce DiffusionDB, the first large-scale text-to-image prompt dataset totaling 6.5TB, containing 14 million images generated by Stable Diffusion, 1.8 million unique prompts, and hyperparameters specified by real users. We analyze the syntactic and semantic characteristics of prompts. We pinpoint specific hyperparameter values and prompt styles that can lead to model errors and present evidence of potentially harmful model usage, such as the generation of misinformation. The unprecedented scale and diversity of this human-actuated dataset provide exciting research opportunities in understanding the interplay between prompts and generative models, detecting deepfakes, and designing human-AI interaction tools to help users more easily use these models. DiffusionDB is publicly available at: https://poloclub.github.io/diffusiondb.
1 Introduction
DiffusionDB addresses the difficulty and unpredictability of prompt writing by providing a large-scale dataset of real Stable Diffusion prompts, images, and hyperparameters. It supports systematic analysis of prompt behavior, model errors, and harmful uses.
- Motivation: Writing prompts for desired image details requires trial and error and can feel random and unprincipled.Users often rely on recurring keywords such as “trending on artstation” and “unreal engine” without understanding why they work.
- Dataset contribution: 6.5TB of data contains 14 million Stable Diffusion images, 1.8 million unique prompts, and hyperparameters specified by real users.The dataset was collected from images shared on the Stable Diffusion public Discord server and released under CC0 1.0.
- Dataset contribution: The dataset includes model hyperparameters alongside images and prompts, supporting analysis of how generation settings relate to outputs.Figure 2 identifies seed, step, CFG scale, sampler, and image size as accompanying hyperparameters.
- Research opportunities: DiffusionDB enables systematic investigation of diverse prompts, associated images, prompt patterns, semantic representations, and model errors.The authors report that particular hyperparameters and prompt styles can lead to model errors.
- Broader impacts: Keyword searches provide evidence of potentially harmful uses of image-generative models, including generating misinformation and nonconsensual pornography.The paper gives examples involving vaccine microchip conspiracies and fabricated Russo-Ukrainian War imagery.
2 Constructing DIFFUSIONDB
The authors construct DiffusionDB by scraping Stable Diffusion Discord activity, linking generated images to prompts and metadata, detecting unsafe content, and distributing the resulting dataset in a modular format. They document both dataset-access safeguards and limitations affecting interpretation.
- Collection and linking: Stable Diffusion Discord messages provide user-generated images, prompts, hyperparameters, random seeds, timestamps, and requester usernames for dataset construction.HTML chat files are parsed to link images with their generation metadata, and collages are split into individual images.
- Identifying NSFW content: NSFW detectors assign scores to prompts and images so researchers can filter potentially unsafe content.Prompt scores use toxicity and sexually explicit probabilities, while image scores combine hentai, sexual, and porn probabilities; blurred images receive score 2.0.
- Identifying NSFW content: 0.3604 precision and 0.9565 recall were obtained for the prompt NSFW detector, while the image detector achieved 0.315 precision and 0.9722 recall.Adjusted recalls were 0.6661 for prompts and 0.3037 for images after correcting for up-sampled positive samples.
- Organizing and distributing DIFFUSIONDB: A modular structure uses 14,000 subfolders of 1,000 images, JSON metadata files, and a column-based Parquet table for flexible subset use and querying.Images receive UUID-based filenames, while metadata includes prompts, generation settings, creator hashes, timestamps, and NSFW scores.
- Organizing and distributing DIFFUSIONDB: The dataset is publicly distributed under CC0 1.0 with creator usernames represented by SHA256 hashes and a reporting form for harmful-content removal.The distribution also provides scripts for downloading and loading the dataset.
3 Data Analysis
The analysis characterizes DiffusionDB’s prompts through length, language, syntactic structure, semantic embeddings, and prompt-image alignment, then identifies hyperparameter- and prompt-related generation errors and potentially harmful uses.
- Prompt Length: 1,819,808 unique prompts were collected from Discord, where users experimented with multiple images and hyperparameters.Prompts were tokenized using Stable Diffusion’s tokenizer, which truncates inputs at 75 tokens.
- Prompt Length: Short prompts of around 6–12 tokens are most popular, while a spike at 75 tokens indicates many exceed the model’s limit.The distribution motivates interfaces that guide users to remain within the token limit.
- Prompt Language: 98.3% of unique prompts are in English, while the dataset includes 34 languages with at least 100 unique prompts.German, French, Italian, and Spanish are the most common non-English languages listed.
- Prompt Syntactic Features: Named entities and noun phrases extracted from comma-separated prompt components are grouped by noun-phrase roots to form phrase hierarchies.The analysis uses these hierarchies to study phrase composition across all 1.8M unique prompts.
- Prompt Syntactic Features: Circle packing encodes phrase frequency by circle size and lets viewers zoom from popular phrases into related noun-phrase subtrees.The visualization surfaces styles such as “digital painting,” “oil painting,” and “portrait painting.”
- Prompt Semantic Features: UMAP visualizations show distinct art and photography prompt regions, with photography splitting into non-human-object and celebrity clusters.Movie-related prompts can appear closer to art and portrait regions in image-embedding space, indicating imperfect prompt-image semantic alignment.
- Stable Diffusion Error Analysis: All four tested hyperparameters—CFG scale, step, width, and height—are negatively correlated with semantically bad images, with p < 0.0001 for each.Negative CFG scales, step = 2, and size (64,512) are shown as concrete failure cases.
- Stable Diffusion Error Analysis: 1.1k unique bad image-prompt pairs remain near-default settings, often involving non-English, very short, or emoji-heavy prompts.These cases have significantly shorter token lengths than the overall prompt set.
4 Enabling New Research Directions
DiffusionDB’s scale, metadata, and paired prompts and images support research directions spanning prompt assistance, image search, model improvement, deepfake detection, and explainable generation.
- Prompt Autocomplete: DiffusionDB enables prompt autocomplete and auto-replace systems that predict likely words, suggest semantic categories, or substitute weaker keywords.Examples include suggesting “depth of field” and “award-winning” or learning replacements from effective prompt patterns.
- Generation through Search: Its 14 million images support generation-through-search systems that retrieve existing images instead of repeatedly running Stable Diffusion.Users can search by natural language or images, while researchers can build structured indexes of images and prompts.
- Improving Generative Models: Stable Diffusion usage logs expose failure modes, user preferences, prompt evolution, and parameter-image chains through joint embeddings and metadata.These signals can inform model enhancement and interfaces for better image-generation experiences.
- Explainable Generation: Similar prompt-image pairs, such as “a happy dog” and “a sad dog,” enable input-permutation analyses for feature attribution in explainable generation.The paired collection supports studying how small prompt changes relate to generated-image differences.
- Deepfake Detection: The dataset’s generated images and metadata can train models to detect synthetic artifacts and classify synthetic images against real images.This direction addresses concerns about deepfakes involving real individuals.
5 Related Work
Prior work studies text-to-image prompting through prompt-engineering research and searchable image collections, while DiffusionDB contributes an open-source dataset of 14 million real prompt-image pairs.
- DiffusionDB contains 14 million real prompt-image pairs, far exceeding the 2k text prompts in PromptSource.
- Text-to-image prompt engineering research spans NLP, computer vision, and HCI, including studies of prompt modifiers and design guidelines.
- Lexica lets users search more than 5 million Stable Diffusion images with prompts but does not release its internal database.
- DiffusionDB differs by publicly releasing an open-source text-to-image prompt dataset for everyone.
6 Conclusion
DiffusionDB is a large-scale, openly released collection of Stable Diffusion images, prompts, and hyperparameters intended to support research and tools for text-to-image generation.
- DiffusionDB contains 14 million Stable Diffusion images with their prompts and hyperparameters collected from the official Discord server.
- The dataset is released under a CC0 1.0 license, and the collection and analysis code is open source.
- The authors intend DiffusionDB to support future development of large generative models and tools that help users use them.
7 Limitations
The authors identify unsafe content, data-source bias, limited image-quality measurement, and uncertain generalizability across generative models as DiffusionDB limitations.
- The dataset uses a limited measure of image quality, and future researchers could recruit annotators to rate its images.
- DiffusionDB includes unsafe images and prompts that escaped the Discord server’s rules, NSFW filter, or moderator removal.The authors compute safety likelihoods and provide reporting and removal mechanisms.
- Discord may be a biased data source, so the dataset’s diverse styles and categories may not represent all users or image-generation practices.
- Findings from DiffusionDB may not generalize to other generative models because effective prompts can differ across models.The paper contrasts comma-separated Stable Diffusion prompts with patterns less common for DALL-E 2 or Midjourney.
8 Ethics Statement
The ethics statement addresses copyright and privacy while noting that the dataset’s broader limitations and impacts are discussed separately in its Data Sheet.
- Privacy: The ethics section identifies privacy as an ethical consideration alongside copyright.
- Copyright: Content shared on the Stable Diffusion Discord server is treated as public domain under users’ agreement to the CC0 1.0 dedication.
- The authors provide a broader discussion of DiffusionDB’s limitations and impacts in a Data Sheet.
Motivation
DIFFUSIONDB was created to address research needs around diffusion models and prompt engineering. It provides real prompt-image pairs for studying model behavior, effective prompts, and image-generation tools.
- DIFFUSIONDB addresses the need to understand how large text-to-image models work and react to prompts.
- The dataset supports research on writing effective prompts for image generation.
- DIFFUSIONDB was created as a large-scale collection of 14 million real prompt-image pairs.
Who created the dataset (e.g., which team,
The dataset was created by Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau at the Georgia Institute of Technology.
- Zijie J. Wang created the dataset.
- Evan Montoya, David Munechika, Haoyang Yang, and Benjamin Hoover also created the dataset.
- The dataset was created at the Georgia Institute of Technology.
Who funded the creation of the dataset?
The dataset was funded in part by a J.P. Morgan PhD Fellowship, NSF grants, DARPA GARD, and gifts from Cisco, Bosch, and NVIDIA.
- Funding included a J.P. Morgan PhD Fellowship and NSF grant IIS-1563816.
- Funding also included DARPA GARD.
- Additional support came from gifts from Cisco, Bosch, and NVIDIA.