Source-linked AI summary
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, Tom Goldstein
TL;DR
Diffusion models may reproduce training content, raising questions about originality and attribution. The paper uses image-retrieval frameworks to compare generated images with training samples and finds replication across dataset scales, including clear Stable Diffusion cases. Its measured Stable Diffusion rate is approximately 1.88% under the study’s search and similarity criteria, but the authors state that replication is underestimated.
Problem
The paper investigates whether diffusion models spontaneously reproduce training images, a question tied to originality and attribution concerns.
Method
The authors benchmark image-retrieval feature extractors and use them to compare generated images with training samples across diffusion models and datasets.
Results
Approximately 1.88% of random Stable Diffusion generations had dataset similarity ≥.5, while replication was frequent for small and medium datasets and seemed undetectable on ImageNet.
Takeaways & Limitations
Replication occurs often enough in diffusion outputs that its presence cannot be safely ignored, including in Stable Diffusion.
Takeaways & Limitations
Stable Diffusion searches covered only 12M LAION images although the model was trained and fine-tuned on much larger image collections, so measured replication is systematically underestimated.
Abstract
from arXiv · showhide
Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes. But do diffusion models create unique works of art, or are they replicating content directly from their training sets? In this work, we study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication. We also identify cases where diffusion models, including the popular Stable Diffusion model, blatantly copy from their training data.
1. Introduction
The paper asks whether diffusion models reproduce training content and develops retrieval-based tools to measure replication across models and dataset properties.
- Diffusion models trained on huge web-scale datasets raise originality, legal, and ethical concerns because data origins are often unknown and models can memorize training data.
- Content replication means reproducing training images partly or wholly, with implications for attribution and the ethical and legal use of diffusion models.
- The authors benchmark image feature extractors on real and synthetic datasets and find that state-of-the-art instance retrieval models work well for detecting replication.
- Replication happens frequently for small and medium datasets, appears undetectable on ImageNet, and is clearly present in Stable Diffusion despite its larger scale.
- Stable Diffusion replication rates are likely underestimated because the search covers only a small subset of its training data, while similarity thresholds remain subjective.
2. Background
The background connects image retrieval and copy detection with research on memorization, inversion, membership inference, and diffusion models.
- Image retrieval searches a database for images containing features from a source image, while inexact copy detection requires high semantic similarity between source and match.
- Self-supervised descriptors, including DINO, provide a basis for image retrieval because they learn representations that match similar images to similar features.
- Recent retrieval systems use vision-transformer backbones, and SSCD specializes in copy detection through self-supervised learning, regularization, and task-specific augmentations.
- Memorization, membership inference, model inversion, and replication are related but distinct: memorization may enable inversion or membership inference without spontaneous generation of training images.
- Diffusion converts Gaussian-noise samples into samples from complex distributions such as natural images.
- Stable Diffusion v1.4 was trained on over 2B images, fine-tuned with 600M images, and searched against only 12M images to manage storage costs.
3. What Counts as Replication?
The paper defines replication around object-level identity rather than subjective style or semantic similarity, while synthetic data illustrate controlled replicated content.
- The study defines replicated content as an object appearing identically in a training image, aside from minor augmentation-related appearance changes.
- Object-level similarity is prioritized because it is more relevant to potential intellectual-property disputes than highly subjective style-wise or semantic similarity.
- Style-based definitions are excluded because they are subjective, typically unrelated to intellectual-property infringement, and unsuitable for images without well-defined style.
- Figure 2 presents synthetic-data constructions as original images, Segmix generation, diagonal outpainting, and patch outpainting.
4. Detecting Content Replication
The paper builds and evaluates retrieval detectors using synthetic and real datasets, local feature metrics, and multiple retrieval models.
- The detection system evaluates 10 feature-extractor prototypes across 10 curated datasets designed to measure replication-detector performance.
- Synthetic datasets create replication through pasted patches, diffusion outpainting, diagonal masking, or augmented and repositioned segmented objects.
- Real retrieval datasets include landmark images and objects photographed under different locations and conditions, providing diverse evaluation settings.
- The model comparison includes retrieval specialists, copy-detection models, and self-supervised methods such as DINO, VICRegL, and MoCo v3.
- Computing the similarity: Global inner products can falsely score an image containing independently copied car and tree content as similar to the unrelated source images together.
- Computing the similarity: Split-product similarity chunks feature vectors, compares corresponding chunks, and takes the maximum, making token-level transformer representations more local.
- Computing the similarity: Split-product is more semantic, whereas inner-product metrics enforce stricter pixel-wise similarity.
- DINO with split-product performs best on average across all 10 datasets; Swin wins on real datasets and SSCD on synthetic datasets.
5. Do Diffusion Models Copy?
Experiments across diffusion models show that smaller training sets produce more direct copying, while larger datasets reduce replication and make generations more similar than identical to training images.
- Smaller training sets tend to produce copied images, and replication decreases as training-set size increases.
- Qualitative observations: Figure 3 compares top-two matches across training-set sizes, showing full replication in the two smaller models but only very similar matches for the whole-dataset model.
- Quantitative observations: Figure 5 compares 10000 generated images with training-data self-similarity using top-1 similarity-score histograms across three training-set sizes.
- Qualitative observations: Models trained on 300 and 3000 Celeb-A images blatantly copy training images, whereas whole-dataset models produce similar but nonidentical generations.
- Quantitative observations: The 300-sample model produces mostly extremely similar generations, while the 3000-sample model produces fewer blatant copies.
- Quantitative observations: For the full-dataset model, generated-to-training similarity histograms strongly overlap training-data self-similarity, indicating no greater average copying.
6. Case Study: ImageNet LDM
The ImageNet latent diffusion case study evaluates copying at both class and population levels using similarity comparisons between generated images and training data. Across 100 classes, similarities remain below the reported copying range, while higher intra-class self-similarity corresponds to higher generation-to-training similarity.
- The study evaluates an off-the-shelf class-conditional latent diffusion model trained on ImageNet at both class and population levels.
- Observations: Across 100 classes, no significant copying is observed; similarity scores never exceed 0.65, and high-similarity matches are similar but not exact copies.
- Observations: Classes with higher average self-similarity have higher maximum generation-to-training similarity, with correlation 0.6 and best-fit slope 0.39.
- Figure 6 plots per-class generation-to-training similarity against training-data self-similarity and compares their score histograms across 100 classes.
7. Case Study: Stable Diffusion
The Stable Diffusion case study finds clear forms of content replication, including verbatim, local, stylistic, and key-phrase-dependent copying. Caption-source images were not typically reproduced, but replication rates and apparent matches depend on retrieval criteria and the searched training subset.
- Experimental setup: Stable Diffusion was evaluated using 9000 LAION Aesthetics 12M source images, their captions, generated images, and top-1 training-set matches.The search used the 12M subset because the model was trained on more than 2 billion images.
- Retrieval analysis: SSCD and DINO often flagged the same images, but DINO returned more diverse matches with lower pixel similarity than SSCD.The visualizations selected images with SSCD similarity > 0.5; SSCD top-10 matches could be identical copies of one training image.
- Content replication: ≈1.88% of random generations exceeded SSCD similarity 0.5, with substantial copying above that threshold.The selected examples included verbatim object-and-background reuse and local background copying.
- Caption-source matching: None of the generations matched their respective caption-source images, although some generations were more representative of the caption than the original source.This indicates that caption sampling did not typically reproduce the image from which the caption was obtained.
- Forms of copying: Stable Diffusion copied pixel-level details, structures, and styles, including known paintings and recognizable wave or sofa forms.Prompt key phrases could trigger exact sofa and wave-shape replications, while artist names frequently elicited known paintings with varying accuracy.
- Caveat: LAION caption sampling may overstate replication rates because key phrases associated with dataset images can conjure memorized images, and no baseline sampler for typical captions exists.Experienced users may also exploit phrases such as “art station,” “35mm,” or artist names.
- Training-image duplication: 34.1 average duplications characterized very close matches (> .5 SSCD), compared with 3.1 for typical matched images and 11.6 for typical random images.Duplicate training images were defined using SSCD score > 0.95; replicated content tended to come from unusually duplicated images.
8. Possible causes of replication
The paper argues that Stable Diffusion’s replication cannot be explained by dataset duplication alone. It instead speculates that text conditioning, skewed repetition, and extensive training updates interact to produce overfitting on part of the data.
- Comparative evidence: Replication is not inevitable: the ImageNet LDM showed no evidence of significant data replication despite similar update rules, training routines, and parameter counts to Stable Diffusion.This comparison rules out several shared architectural and training factors as sufficient explanations for their behavioral difference.
- Dataset duplication: Small-dataset models consistently showed strong replication even when duplicated images were unlikely, so image duplication alone is an incomplete explanation.With the same number of training updates, duplicating every unique image 1000 times would yield the same model as using the unique dataset.
- Proposed factors: The authors speculate that Stable Diffusion’s replication reflects an interaction among text conditioning, skewed image repetitions, and enough gradient updates to overfit a data subset.These are proposed contributors rather than an established causal account.
9. Limitations & Conclusion
The study finds that diffusion models can reproduce high-fidelity training content, but its Stable Diffusion search covers only a small fraction of the model’s training data and therefore underestimates replication.
- Diffusion models can reproduce high-fidelity content from their training data.
- The Stable Diffusion search covered 12M images, although the model was trained on over 2 billion images and later fine-tuned on 600M images.
- The searched dataset comprised less than 0.6% of the total training data.
- Replication results systematically underestimate the amount of replication because some sources fall outside the searched split and some copies evade retrieval.
Models
The appendix documents prompts, training configurations, and supplementary figure materials for the paper’s diffusion-model experiments.
- The appendix provides prompts used for Stable Diffusion figures and additional figures accompanying the main-paper observations.
- Figure 1 materials include the source caption and match caption for each displayed comparison.
- The appendix lists prompts for Figure 10 and manually curated prompts for studying copying of style versus content.
- The Celeb-A models were trained with dataset sizes of 300, 3000, and the full dataset using specified epochs, batch sizes, learning rates, and warmup steps.
- Oxford Flowers models used 100-image, 1000-image, and full-dataset training configurations with distinct epochs, batch sizes, learning rates, and warmup steps.
- Stable Diffusion generations used 9000 sampled captions, one generation per caption, guidance scale 7.5, strength 0.5, and 50 steps.
C.1. Misc results
Supplementary results show retrieval comparisons for Celeb-A and ImageNet generations, alongside caption examples and manually curated prompts for examining content and style copying.
- Supplementary figures show Celeb-A generations and their top five DINO split-product matches across models trained on 300, 3000, and 30k images.
- ImageNet supplementary figures show conditional LDM generations alongside their closest or top four matches from the ImageNet training set.
- The caption pair contrasts “Golden Globes best fashion on the red carpet” with “A photo of Red Carpet Wedding Dress Inspiration from the Golden Globes.”
- Figure 13 presents the generation caption and the caption of the top match for expanded teaser examples.
- The supplementary materials include 20 manually curated prompts for examining copying of style versus content, with six used for Figure 8.