PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting Training Data from Diffusion Models

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, Eric Wallace

arXiv:2301.13188v1cs.CRcs.CVcs.LG

TL;DR

Diffusion models are often treated as privacy-preserving generators, but the paper tests whether they memorize and regenerate individual training images. Using extraction attacks and controlled model studies, it finds substantial memorization, greater leakage than GANs, and unresolved privacy challenges.

  • Problem

    Diffusion models were assumed to generate synthetic images without memorizing training data, yet the privacy implications of that assumption remained uncertain.

  • Method

    The authors define image memorization, develop a two-stage generate-and-filter extraction attack, and train hundreds of models to study privacy across modeling and data choices.

  • Results

    State-of-the-art diffusion models memorize and regenerate training images, leak 2× more than comparable GANs, and yield over 70% TPR at 1% FPR in a baseline attack.

  • Takeaways & Limitations

    Synthetic data from diffusion models does not provide privacy for free; deduplication and minimizing over-training are recommended, but simple defenses are insufficient.

  • Takeaways & Limitations

    The memorization definition is conservative and excludes recognizable images that are not near-identical reconstructions, although such outputs may still cause privacy harms.

Abstract

from arXiv · show

Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images. In this work, we show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models, ranging from photographs of individual people to trademarked company logos. We also train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy. Overall, our results show that diffusion models are much less private than prior generative models such as GANs, and that mitigating these vulnerabilities may require new advances in privacy-preserving training.

1 Introduction

Diffusion models were widely assumed to generate novel images without memorizing training data, but this work demonstrates that state-of-the-art models can regenerate individual training examples. The authors develop an extraction attack and find substantial privacy risks, including personally identifiable photos and trademarked logos.

  • Motivation: Diffusion models were assumed not to memorize or regenerate training data, an assumption that underlies proposed privacy guarantees for synthetic images.If false, memorization would undermine those guarantees and raise concerns about model generalization and digital forgery.
  • Contribution: The authors show that state-of-the-art diffusion models memorize and regenerate individual training examples.
  • Method: A two-stage generate-and-filter attack flags generations using membership inference scoring criteria to extract near-identical training images.The method is applied to Stable Diffusion and Imagen.
  • Findings: The extracted images include personally identifiable photographs and trademarked company logos.
  • Findings: Diffusion models leak more than twice as much training data as GANs and existing privacy-enhancing techniques do not provide an acceptable privacy-utility tradeoff.The authors analyze these effects by training hundreds of diffusion models under varied modeling and data settings.

2 Background

Diffusion models generate images by iteratively denoising random noise, extending a simple training objective into a scalable and controllable image-generation process. The background also situates their privacy risks within broader neural-network attacks and concurrent diffusion-model research.

  • Diffusion models: Diffusion models have displaced GANs and VAEs in high-resolution image generation because they produce higher-quality samples and are easier to scale and control.
  • Diffusion models: Diffusion models train image denoisers by adding Gaussian noise to clean images at sampled time steps and predicting the noise for removal.The noised image is formed as x′ ← √a_t x + √(1−a_t) ε.
  • Diffusion models: Generation starts from random Gaussian noise and iteratively applies the denoiser to produce a final image resembling a natural image.The process uses a noise schedule and repeatedly transforms z_t into z_t−1.
  • Conditioning: Class-conditional and text-conditioned models guide generation with labels or text embeddings, such as prompts describing desired image content.
  • Scope: The paper’s simplified diffusion-model description omits significant technical details because the authors regard them as orthogonal to the attack results.
  • Training-data privacy attacks: The paper studies privacy risks beyond membership inference, including inversion, attribute inference, and extraction attacks that can recover training examples.Concurrent work examined membership inference and non-adversarial memorized samples, whereas this work studies stronger extraction risks.

3 Motivation and Threat Model

The paper motivates memorization research through privacy, copyright, and generalization concerns, then defines adversaries that query or control diffusion generators to recover training data. Its ethics scope relies primarily on publicly available images and advance disclosure to model developers.

  • Motivation: Memorized outputs can create privacy and copyright risks, especially as diffusion models may be applied to sensitive domains such as medical imagery.
  • Motivation: Claims that diffusion models protect the privacy or usage rights of real images are challenged by the possibility that the models regenerate training data.The paper concludes that diffusion models may be unfit for this privacy-preserving purpose.
  • Motivation: Studying memorization also informs whether large-scale generative models produce novel images or directly copy and remix training data.
  • Threat model: The threat model includes black-box adversaries that query generators and white-box adversaries that control randomness and denoise arbitrary inputs.
  • Threat model: For conditional generators, the adversary is assumed to know captions for some training images, representing a worst-case privacy risk.
  • Adversarial goals: The strongest adversarial goal is data extraction: recovering a training image that is almost identical to an image in the training set.The paper also considers data reconstruction and membership inference as related goals.
  • Ethics and broader impact: The ethics analysis focuses on models trained on publicly available images and avoids displaying unintended private material, while acknowledging that online availability does not imply intended public release.The authors also shared an advance copy with model developers to support safeguards and software changes.
  • Ethics and broader impact: The paper frames disclosure of these vulnerabilities as a way to preempt future harms and encourage responsible diffusion-model training.

4 Extracting Training Data from State-ofthe-art Diffusion Models

The paper defines image memorization through approximate extraction and applies a generate-and-filter attack to Stable Diffusion and Imagen. It finds near-copies of training images, with extraction strongly associated with duplication and involving personally identifiable, commercial, and copyrighted content.

  • 4.2 Extracting Data from Stable Diffusion: The attack generates images with standard sampling, then uses membership-inference criteria and clique-finding to identify groups of near-identical generations.For each prompt, 500 generations are connected when sufficiently similar; a largest clique of at least 10 generations is predicted to represent memorization.
  • 4.1 Defining Image Memorization: Approximate memorization is defined by whether an efficient algorithm can generate an image within a distance threshold of a training example.The paper uses normalized Euclidean 2-norm distance as its default similarity measure and defines Eidetic Memorization by limiting the number of near-duplicates in the dataset.
  • 4.1 Defining Image Memorization: The definition is intentionally conservative: recognizable new images of individuals do not count unless they are near-identical reconstructions of particular training images.For example, a generated Obama image is excluded because its nearest training images all have ℓ2 distance above 0.3.
  • 4.2.2 Extraction Results: 109 near-copies appear among the top 1,000 generated images after manual inspection, including 13 that fall outside the paper’s 2-norm definition.The extracted images include near pixel-perfect replicas with ℓ2 difference below 0.05.
  • 4.2.2 Extraction Results: 50 memorized images are identified from 175 million generations with 0 false positives, while all memorized images are extracted at precision above 50%.The paper evaluates the attack using precision-recall curves under both its distance-based and manual memorization definitions.

5 Investigating Memorization

Controlled experiments show that diffusion models memorize and expose CIFAR-10 training images through untargeted extraction, membership inference, and inpainting attacks. Adaptive similarity scoring and combined attacks substantially improve detection, while privacy leakage increases with model quality.

  • 5.1 Untargeted Extraction: An uncalibrated ℓ2 threshold fails because low absolute distances can occur for non-memorized images, whereas unusually low per-image distances identify extractions.A failed example has nearest distance 0.06, comparable to many training images; a successful example has distance 0.07, while most others exceed 0.2.
  • 5.1 Untargeted Extraction: 1,280 unique extracted images, or 2.5% of CIFAR-10, were identified using an adaptive nearest-neighbor similarity attack.The attack compares a generated image with its nearest training image relative to distances from nearby training examples, using α = 0.5 and n = 50.
  • 5.2.1 Baseline Attack Results: Over 70% TPR at 1% FPR was achieved for membership inference, compared with below 20% TPR at 1% FPR for state-of-the-art classifiers.Evaluating the diffusion loss at timesteps t ∈ [50,300] produced the strongest attacks.
  • 5.2.1 Baseline Attack Results: Combining the attack strategies increased TPR at 0.1% FPR by more than sixfold, from 7% to 44%.The components include additional Monte Carlo samples and horizontal-flip augmentation.
  • 5.2.2 Memorization and Model Quality: Higher diffusion-model quality was associated with greater privacy leakage, suggesting that stronger future models may be less private.The analysis evaluated attack success as a function of FID during training.
  • 5.3 Inpainting Attacks: Inpainting provides a targeted attack by masking part of an image and using the diffusion model to reconstruct the missing region.The method compares reconstruction efficacy for training and testing images, using membership inference to rank reconstructions.

6 Comparing Diffusion Models to GANs

The paper compares diffusion models with GANs using membership inference and black-box extraction attacks. Across these evaluations, diffusion models leak more training data than GANs, including when the models achieve similar generative quality.

  • Membership Inference: Diffusion models show higher membership-inference leakage than GANs, reaching 50% TPR at 0.1% FPR versus below 30% TPR for GANs.The comparison uses loss-threshold and LiRA attacks on GAN discriminators across BigGAN, MHGAN, and StyleGAN.
  • Data Extraction: Diffusion models memorize more training data than GANs, even when GANs reach similar FID; the best DDPM memorizes 2× more than StyleGAN-ADA.The paper also reports that StyleGAN-ADA memorizes 3× more images than the weakest GANs as FID improves.
  • Overall Comparison: Diffusion models are less private than GANs under the evaluated default training configurations.This conclusion is supported by both membership-inference and near-copy extraction comparisons.
  • Shared Memorization: 244 images were memorized by both a diffusion model and a StyleGAN, versus an expected overlap of 10 under uniform random memorization.The reported overlap has p < 10^-261, indicating that the shared images are highly non-random under the paper’s model.

7 Defenses and Recommendations

The paper evaluates deduplication, differential privacy, and canary auditing as ways to reduce or measure memorization. Deduplication is imperfect, DP-SGD caused training failures in these experiments, and canaries provide a lighter-weight audit signal.

  • Deduplication: Deduplication is not a perfect solution for reducing diffusion-model memorization.The experiment removes 5,275 similar images from CIFAR-10 and retrains a diffusion model to evaluate the intervention.
  • Differential Privacy: DP-SGD caused CIFAR-10 diffusion-model training to diverge consistently, even at ε around 50.Applying gradient clipping or noise independently also caused training to fail, and the authors leave further investigation to future work.
  • Auditing: Membership-inference audits can be expensive because they require training many shadow models, motivating lighter-weight alternatives such as canary insertion.The paper presents canary exposure as an empirical way to audit memorization in diffusion models.
  • Canary Auditing: Canary exposure reached the maximum value of 10 for some canaries after only two insertions.Exposure was not strictly increasing with duplicate count, possibly because some canaries were harder than others.

8 Related Work

Prior work studies memorization across language and image-generative models using different architectures, similarity notions, and threat models. This paper positions its contribution as a broader privacy analysis of image diffusion systems.

  • Memorization in Language Models: Language-model studies use two-step extraction attacks resembling the paper’s approach, while this work focuses on image generation and semantic regeneration.The related work spans multiple domains, architectures, and threat models.
  • Memorization in Image Generation: Earlier image-generation research examines training-data similarity, mode collapse, individual-sample effects, and why GANs may replicate examples.These studies primarily analyze memorization through generalization and novelty in GANs.
  • Recent and Concurrent Work: Concurrent work reports privacy risks for StyleGAN faces and semantically similar Stable Diffusion outputs, whereas this paper covers Imagen, CIFAR models, and membership inference.The paper distinguishes its scope by combining more systems and threat models than the cited works.

9 Discussion and Conclusion

The paper finds that diffusion models memorize and regenerate training images, creating extraction and privacy risks that existing defenses do not adequately resolve. It recommends stronger auditing and privacy protections while emphasizing open questions about generalization and human-aligned definitions of memorization.

  • Diffusion models memorize and regenerate individual training images, enabling adversaries to launch training-data extraction attacks.
  • 2× more training data is memorized by state-of-the-art diffusion models than by comparable GANs, while more useful diffusion models memorize more than weaker ones.
  • The paper leaves open whether large-scale models generate novel outputs or copy and interpolate between training examples.
  • A more comprehensive analysis is needed because the study’s simple ℓ2 measure may not capture nuanced, human-aligned notions of data copying.
  • Deduplicating training data and minimizing over-training are recommended as imperfect defenses, alongside auditing models for privacy risk.
  • The paper recommends practical privacy-preserving techniques when available and warns that synthetic data does not provide privacy automatically.

Contributions

The contributions span the problem formulation, experiments on diffusion and GAN models, and the presentation of qualitative extraction examples. The work combines large-model extraction studies with CIFAR-10 analyses of memorization and privacy.

  • Nicholas, Jamie, Vikash, and Eric independently proposed the problem of extracting training data from diffusion models.
  • Preliminary experiments identified cases of data extraction in diffusion models.
  • Experiments on Stable Diffusion and Imagen included counting duplicates in the LAION training dataset.
  • Membership-inference, inpainting, and diffusion-extraction experiments were conducted on CIFAR-10 models.
  • Experiments examined canary memorization and memorization in GANs, including extraction from pretrained GANs.
  • The paper includes qualitative examples collected across its figures, with figures covering Stable Diffusion and extracted CIFAR-10 images.

C.1 Membership Inference at Different Training Steps

Membership-inference vulnerability increases with training exposure and can be amplified by reducing attack noise. Conditional and unconditional models show similar utility, with the conditional model marginally more vulnerable.

  • More than 2000 training exposures make membership attacks invariably perfect for an example.
  • Membership-inference attacks improve when loss is averaged over multiple noise samples or evaluated on augmented candidate images.
  • The hardest-to-attack CIFAR-10 examples are duplicates, whereas the easiest are visually out-of-distribution examples.
  • Conditional and unconditional models reach approximately the same FID, between 3.5-4.2 FID, while the conditional model is marginally more vulnerable.
  • Figure 21 compares membership attacks against conditional and unconditional CIFAR-10 diffusion models.

D More Inpainting Attacks on CIFAR-10

The inpainting attack generates many reconstructions, ranks them with membership-inference losses, and uses contrastive scoring to identify memorized training images. The attack succeeds for included images but fails for images absent from training.

  • For an image included in training, smaller contrastive losses correspond more strongly to smaller ℓ2 distances from the target.
  • Contrastive loss strengthens the relationship between reconstruction loss and ℓ2 distance by dividing main-model loss by support-model loss.
  • The attack fails when the target image is absent from training, and contrastive loss does not significantly increase the Pearson correlation coefficient.
  • The CIFAR-10 GAN comparison used 256 models for each GAN architecture, with BigGAN trained for 200,000 steps to increase image fidelity.

F Additional GAN Extraction Results

Figures 24 and 25 provide additional examples of training-data extraction from GANs trained on CIFAR-10 and from publicly available GAN and diffusion models.

  • Figures 24 and 25 contain additional training examples extracted from GANs trained on CIFAR-10.
  • Figure 24 shows extracted CIFAR-10 training examples across architectures and 107 generations.
  • Figure 25 compares real images with closely matching synthetic images using normalized ℓ2 pixel-space distance across pretrained GANs and DDPM models.
  • For StyleGAN-ADA and DDPM, Figure 25 displays 120 image pairs with the smallest normalized ℓ2 distances.
  • The Figure 25 comparison uses 1M generations and displays all memorized training images for the other models.
Loading 2301.13188v1…