Source-linked AI summary
Understanding and Mitigating Copying in Diffusion Models
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, Tom Goldstein
TL;DR
Diffusion models can replicate training images, and the paper examines whether image duplication alone explains this behavior. It analyzes replication in real and controlled settings, finding that caption conditioning is a fundamental factor and proposing caption-diversification strategies to mitigate copying. The authors recommend evaluating these strategies in specific use cases because other factors may also influence outcomes.
Problem
The paper asks what mechanisms drive replication in text-to-image diffusion models beyond duplicated training images, a concern because generated images can reproduce training data.
Method
The paper studies replication using real user prompts and controlled fine-tuning experiments, then tests randomized and augmented caption-conditioning strategies during training or inference.
Results
Caption conditioning and caption diversity are fundamental replication factors: diversified captions can reduce memorization even when image duplication increases, while image duplication alone is not a good predictor.
Takeaways & Limitations
Mitigating copying requires attention to both image duplication and text conditioning, with caption randomization and augmentation providing proposed training- or inference-time strategies.
Takeaways & Limitations
Other factors may influence replication outcomes, so the proposed mitigation strategies should be evaluated within specific use cases before production deployment.
Abstract
from arXiv · showhide
Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models. While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set.
1 Introduction
The paper analyzes memorization in modern diffusion models and finds that text conditioning is a major driver of data replication alongside image duplication. It proposes caption-randomization strategies to mitigate copying during training and inference.
- Image duplication alone cannot explain much of the replication observed at test time.
- Text conditioning plays a major role in data replication.
- Diversifying captions can greatly mitigate test-time replication even when training images remain highly duplicated.
- The paper proposes randomizing text conditioning during training or testing to reduce replication.
2 Related work
Prior work has established empirical evidence that generative models can memorize or reproduce training data, but the mechanisms behind diffusion-model memorization and effective inhibition remain uncertain. Existing mitigation approaches include input filtering, although such filters can be circumvented.
- Most existing insights into generative-model memorization are empirical, spanning GANs and generative language models.
- Prior diffusion-model studies found that 0.5-2% of generated images were partial object-level duplicates of training data.
- The mechanisms causing memorization in diffusion models and ways to inhibit it remain uncertain aside from recent theoretical work on image-duplication copyright issues.
- Diffusion-model mitigations have focused on input filters that can be repurposed to block known copyrighted data.
- Simple input filters can be easily circumvented.
3 How big of a problem is data replication?
The paper measures replication under real-world user prompts rather than captions sampled directly from the training dataset. In Stable Diffusion 2.1, approximately 1.2% of generated images crossed the similarity threshold for possible duplication, though the method likely underestimates the true rate.
- Prior studies amplified replication by prompting diffusion models with captions sampled directly from LAION, whereas this work uses real-world user-submitted prompts.
- The study generates images from 100K randomly sampled user-generated captions and searches for closest matches across approximately 400 million training images.
- ∼1200 images (1.2%) exceeded a similarity score of 0.5, indicating they may be duplicates.
- The estimated replication rate is likely an underestimation because SSCD-based search may miss possible matches.
- Some SSCD matches were false positives associated with simple textures in generated images.
4 Experimental Setup
The experiments fine-tune Stable Diffusion models on curated and sampled datasets, using controlled training settings and metrics for quality and memorization. The setup compares image and caption duplication while measuring image similarity and output quality.
- The study focuses on fine-tuning large pretrained diffusion models on smaller datasets to keep experimental costs tractable.
- The datasets include Imagenette and randomly sampled LAION-2B subsets of 10,000 and 100,000 images, with captions available for the LAION subsets.
- Stable Diffusion v2.1 initializes the experiments; unless otherwise noted, only the U-Net is fine-tuned while text and autoencoder components remain frozen.
- FID evaluates generated-image quality and diversity by comparing generated and training-set distributions, with lower scores indicating better quality and diversity.
- SSCD similarity above 0.5 indicates strong visual similarity and likely partial object-level copying from the top-1 training-image match.
- Dataset similarity is defined as the 95-percentile of image-level similarity scores to focus on the highest-scoring 5% of generated images.
5 Data Duplication is Not the Whole Story
Data duplication contributes to replication, but it cannot explain test-time memorization by itself. Caption conditioning strongly affects copying: diverse, content-correlated captions increase memorization, while partial image-caption duplication mitigates it.
- Duplication and replication: Data replication has been observed even in models trained on de-duplicated datasets, so duplication alone cannot explain test-time memorization.Stable Diffusion v2.1 showed little replication for identified duplicate clusters but still exhibited memorization when scanning a larger dataset portion.
- Duplication and replication: Higher duplication generally increases dataset similarity, although the relationship is non-monotonic for LAION-10k at high duplication factors.For LAION-10k, similarity rises until ddf=10 and then decreases; FID can improve with duplication, potentially when the dataset is memorized.
- Caption conditioning: Caption conditioning changes memorization substantially: fixed captions yield the least memorization, while BLIP captions yield the most.Random captions do not maximize memorization, indicating that captions need meaningful correlation with image content to retrieve specific images.
- Caption conditioning: Training the text encoder increases similarity across conditioning types, especially for BLIP and random captions, supporting captions as specific retrieval keys.The effect is strongest when captions are highly specific or unique.
- Image-caption duplication: Partial duplication consistently produces less memorization than full duplication, even as duplication rates increase.This reduction comes with a moderate FID increase at higher duplication levels.
6 Effect of the Training Regimen
The training regimen affects memorization through training duration, data quantity, and image complexity. Longer exposure and dataset properties create a trade-off between model quality and copying, with simpler images more prone to memorization.
- Training length: Image-and-caption duplication produces the highest similarity across training epochs, while image-only duplication remains consistently lower.The comparison suggests that caption diversity for duplicated images strongly affects memorization during longer training.
- Image complexity: LAION-10k spans a wider range of image complexity than Imagenette, while Imagenette is predominantly composed of higher-complexity real-world images.Complexity is measured using histogram entropy and JPEG compressibility.
- Image complexity: -0.32/-0.29 correlations link similarity to entropy and compression metrics, respectively, with p-values of 8e-98/7e-80.The correlations are statistically significant and are reported for LAION-10k models without duplication.
- Image complexity: Images memorized without training-data duplication are likely to be simple in structure.High- and low-similarity populations show clear separation across both complexity measures.
- Training regimen: Training length, data quantity, and setup must balance model quality against memorization.The paper frames regimen selection as a trade-off rather than a single universally optimal setting.
7 Mitigation strategies
The paper mitigates copying by randomizing or augmenting captions during training and inference. Training-time strategies are more effective, while inference-time methods can be retrofitted into existing models and overall reduce copying behavior.
- Mitigation strategies: Caption randomization and augmentation are studied as training- and inference-time strategies for reducing data replication.The strategy family includes multiple captions, embedding noise, caption replacement, token replacement or addition, and word repetition.
- Training versus inference: Training-time mitigations are more effective, whereas inference-time mitigations are easier to retrofit into existing diffusion models.The paper evaluates both approaches qualitatively and quantitatively.
- Evaluation: Table 1 compares similarity scores under no duplication, image duplication (ddf=5), and image-and-caption duplication (ddf=5), where lower similarity is better.Best training- or test-time results are highlighted in the table.
- Effectiveness: The evaluated mitigation strategies are quite effective at reducing copying behavior, including in large modern diffusion models.Qualitative comparisons use finetuned models for training mitigations and Stable Diffusion directly for inference mitigations.
8 Recommendations for a safer model
The paper recommends addressing copying before, during, and after training by detecting duplication and diversifying captions, while evaluating mitigation strategies for specific deployments.
- Recommendations: Caption diversity can reduce replication even when images remain highly duplicated, making conditioning a central mitigation target.The paper identifies conditioning and caption diversity as fundamental factors alongside image deduplication.
- Before training: Before training, identify and collapse image-overlap clusters, resample duplicate captions, and hand-curate acceptable repetitions such as famous paintings.The proposed workflow uses SSCD-style copy detection, caption resampling, and human judgment for concept-specific exceptions.
- During training: During training, resampling captions for duplicated images is reported as the most effective way to reduce copying.Near-duplicates need not be removed entirely if their captions are resampled.
- After training: After training, inference-time mitigations can reduce copying through user-triggered resampling or rejection sampling against known duplicate clusters.These strategies can be applied after a model has already been trained.
- Evaluation: The authors caution that other factors may influence memorization and recommend evaluating mitigation effectiveness within each intended use case before production deployment.The recommendation preserves the paper’s scope rather than treating text conditioning as the only determinant.
D Extended experimental settings
The experiments generally use 100k training iterations, batch size 16, Adam optimization, one RTX-A6000 per model, and substantial GPU time for training and inference.
- Training setup: Models are generally trained for 100k iterations with batch size 16 using Adam, β1 = 0.9, β2 = 0.999, and weight decay = 1e−2.The stated setup uses one RTX-A6000 per model and approximately 24 hours of training.
- Compute: Inference uses RTX-A5000 hardware and takes approximately 5 hours to generate enough samples for metric computation.The study reports approximately 5000 GPU hours for training and another 1000 GPU hours for inference-related analysis.
E Why do diffusion models copy? Extended Results
Extended experiments show that replication depends on both image duplication and conditioning, with caption similarity, training duration, text-encoder training, and dataset properties shaping similarity and FID outcomes.
- Model conditioning: Higher conditioning-caption uniqueness increases similarity scores, while BLIP caption conditioning produces the highest similarity and lowest FID among the reported conditioning models.The same FID trend is observed for fine-tuned models in Figure 14.
- Duplication: Higher duplication levels increase both similarity and FID on LAION-10k and Imagenette, but image-only and image-plus-caption duplication differ across datasets.The difference is attributed to diverse LAION-10k captions versus similar Imagenette captions.
- Training regimen: Longer training consistently raises similarity and FID, while image duplication yields lower similarity than image-plus-caption duplication on both datasets.The comparison covers no duplication, image duplication, and image-plus-caption duplication.
- Training regimen: Training the text encoder increases similarity scores while reducing FID in both LAION-10k and Imagenette.Similar trends persist with image-plus-caption duplication at ddf=5 across conditioning types.
- Image complexity: LAION-10k has lower self-similarity and slightly simpler complexity distributions than Imagenette, while similarity correlates significantly with training-data complexity across tested scenarios.The correlation is reported for models without duplication, with image duplication, and with image-plus-caption duplication.
- Mitigation trade-offs: Gaussian Noise compromises FID, whereas random token or number addition during inference compromises CLIP score most among the reported mitigation strategies.The corresponding FID and CLIP evaluations complement the similarity results in Table 1.
F.3 Extended Qual results
The extended qualitative results use fixed generation settings to compare inference prompts and train-time mitigation outputs, with Multiple Captions reported as consistently effective.
- Inference settings: Inference examples use Stable Diffusion 1.4 with seed 2, guidance scale 7.5, and 50 inference steps.Table 5 contrasts copying-inducing prompts with prompts modified by Random Token mitigation.
- Qualitative findings: Multiple Captions works consistently in the extended train-time examples, whereas the other methods are not always effective.The authors report this pattern across the qualitative results shown in Figure 16.
- Qualitative comparison: The qualitative figure compares training images, regular generations, Stable Diffusion outputs, and generations from five train-time mitigation strategies.The strategies are Multiple Captions, Gaussian Noise, Random Caption replacement, Random Token replacement and addition, and Caption Word Repetition.