PaperlayerView exact revision on arXiv

Source-linked AI summary

Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models

Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng, Ben Y. Zhao

arXiv:2310.13828v3cs.CRcs.AI

TL;DR

Massive text-to-image training datasets are widely assumed to make data poisoning infeasible because attacks would require millions of samples. The paper introduces Nightshade, an optimized prompt-specific attack using visually benign poison images, and finds that it can corrupt targeted prompts with fewer than 100 samples, spread to related concepts, and destabilize general model behavior.

  • Problem

    Poisoning text-to-image models is commonly thought to require millions of samples because their training datasets contain hundreds of millions or billions of images.

    Source: §1 · p. 1

  • Method

    Nightshade optimizes visually benign image–text poison pairs to target specific prompts without requiring access to the model’s internal training or generation pipeline.

    Source: §1 · pp. 1–2, §5.1 · p. 6, §5.2 · p. 6

  • Results

    Nightshade can corrupt a Stable Diffusion SDXL prompt with fewer than 100 poison samples, bleed into related concepts, compose across attacks, and destabilize general features.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Nightshade could provide content owners a last-resort defense against web scrapers that ignore opt-out and do-not-crawl directives.

    Source: §1 · p. 2

    The paper leaves the cause of whole-model degradation from composable attacks for future work and notes that effective detection would need to scale to millions or billions of samples.

    Source: §6.5 · p. 11, §8 · p. 14

Abstract

from arXiv · show

Data poisoning attacks manipulate training data to introduce unexpected behaviors into machine learning models at training time. For text-to-image generative models with massive training datasets, current understanding of poisoning attacks suggests that a successful attack would require injecting millions of poison samples into their training pipeline. In this paper, we show that poisoning attacks can be successful on generative models. We observe that training data per concept can be quite limited in these models, making them vulnerable to prompt-specific poisoning attacks, which target a model's ability to respond to individual prompts. We introduce Nightshade, an optimized prompt-specific poisoning attack where poison samples look visually identical to benign images with matching text prompts. Nightshade poison samples are also optimized for potency and can corrupt an Stable Diffusion SDXL prompt in <100 poison samples. Nightshade poison effects "bleed through" to related concepts, and multiple attacks can composed together in a single prompt. Surprisingly, we show that a moderate number of Nightshade attacks can destabilize general features in a text-to-image generative model, effectively disabling its ability to generate meaningful images. Finally, we propose the use of Nightshade and similar tools as a last defense for content creators against web scrapers that ignore opt-out/do-not-crawl directives, and discuss possible implications for model trainers and content creators.

1. Introduction

The paper challenges the assumption that poisoning massive text-to-image training datasets requires millions of samples, showing that concept-specific attacks can succeed with few optimized poisons. It introduces Nightshade as a stealthy, potent attack with effects that extend across related concepts and can destabilize model behavior.

Contributions

  • 50 optimized Nightshade samples can make Stable Diffusion SDXL generate cows for prompts containing “car.”The attack targets a specific prompt concept without requiring access to the model’s internal pipeline.Source: §1 · p. 1

Motivation

  • Training data for individual concepts is often only on the order of thousands despite models being trained on millions or billions of images.This concept sparsity makes prompt-specific poisoning feasible because attackers need to offset only concept-associated training data.Source: §1 · p. 1

Initial Attack

  • A proof-of-concept mislabeling attack corrupted concepts using 500–1000 poison samples, but remained detectable and required too many samples.The attacks were evaluated on SDXL using CLIP-based classification and an IRB-approved user study.Source: §1 · pp. 1–2

Nightshade

  • Nightshade creates visually benign-looking poison images through optimization, enabling stronger attacks with roughly 100 samples while avoiding human and prompt-based detection.The optimized samples are shifted in feature space but remain visually similar to their benign counterparts.Source: §1 · p. 2
  • Nightshade effects bleed into related concepts, multiple attacks can be triggered by one prompt, and about 250 attacks can corrupt basic model features so meaningful images are no longer generated.The paper also reports transferability across models and resistance to several defenses.Source: §1 · p. 2

Implications

  • The paper proposes Nightshade as a last-resort tool for content owners when opt-out and do-not-scrape directives are ignored.It frames the tool as a potential disincentive against unauthorized training by model developers.Source: §1 · p. 2

Related Work

  • Nightshade differs from Glaze and Mist because it targets the base model for all users rather than local fine-tuning on a small protected image set.Glaze and Mist assume that most training images are protected, whereas Nightshade must overcome large amounts of benign training data.Source: §1 · p. 2

2. Background and Related Work

Text-to-image systems use large, continuously updated web-scraped datasets, creating a setting where poisoned image–text pairs could enter training pipelines. Prior diffusion attacks rely on privileged pipeline access or target local fine-tuning, whereas this work assumes only the ability to add training data.

2.1. Text-to-Image Generation

  • Latent diffusion reduces training cost by operating in a lower-dimensional feature space, enabling models to train on datasets containing 500 million to 5 billion web-scraped image–text pairs.Model developers commonly continue training existing models with newly collected data because training from scratch is expensive.Source: §2.1 · p. 2
  • The paper considers both training from scratch and continuous updating of an existing model with additional data.These are presented as the two prevalent real-world training scenarios.Source: §2.1 · p. 2

2.2. Data Poisoning Attacks

  • Prior poisoning research includes classifier backdoors, clean-label attacks, poison detection, and robust training defenses, but adaptive attacks continue to challenge defenses.Backdoor attacks typically use hidden triggers to induce targeted inference-time behavior.Source: §2.2 · p. 2
  • Existing diffusion backdoor attacks assume attackers can modify denoising steps or the overall training loss.Those assumptions provide access that the present threat model does not grant.Source: §2.2 · p. 3
  • This work targets ordinary prompts without triggers and assumes attackers can only add poison image–text pairs, with no access to training or deployment pipelines.The attack can therefore target everyday concepts such as objects or styles under a more limited threat model.Source: §2.2 · p. 3
  • Glaze and Mist disrupt local fine-tuning using 10–20 protected images, whereas this work seeks to corrupt the general-purpose base model with a small optimized poison set.Adapting those tools to prompt-specific poisoning produced poor attack performance.Source: §2.2 · p. 3

3. Feasibility of Poisoning Diffusion Models

The paper argues that prompt-specific poisoning is feasible because attacks need to overcome sparse data associated with a targeted concept rather than the entire training corpus. It defines a web-scraping threat model and measures long-tailed word and semantic sparsity in LAION-Aesthetic.

3. Feasibility of Poisoning Diffusion Models

  • The attack targets one or more concepts, defined as keywords such as “dog” or “anime” that condition image generation.The attack does not require modifying the training pipeline or diffusion process.Source: §3 · p. 3

3.1. Threat Model

  • Attackers need only inject poison image–text pairs and have no access to model training or deployment pipelines.The evaluation covers both training from scratch and continuous updating of pretrained models.Source: §3.1 · p. 3

3.2. Concept Sparsity Induces Vulnerability

  • Conventional poisoning estimates require more than 5% of data for classifier backdoors, 20% for indiscriminate attacks, and 50% for one diffusion attack, making them impractical at web-scale.Poisoning 1% of datasets with hundreds of millions or billions of samples would require millions to tens of millions of images.Source: §3.2 · pp. 3–4
  • Concept sparsity means that individual concepts and their semantic neighborhoods occupy only a small, uneven fraction of the training corpus, allowing poisons to offset their clean contributions.Examples include 0.1% of samples for “dog” and 0.04% for “fantasy.”Source: §3.2 · p. 4

3.3. Concept Sparsity in Today’s Datasets

  • In LAION-Aesthetic, over 92% of concepts occur in fewer than 0.04% of images, with a mean frequency of 0.07% across ten commonly used concepts.The dataset contains 600 million image–text pairs and 22,833 valid English words.Source: §3.3 · p. 4
  • Figure 2 presents word and semantic concept frequencies as long-tail distributions with logarithmic y-axes.The figure summarizes sparsity at both exact-word and related-concept levels.Source: Figure 2 · p. 4, §3.3 · p. 4
  • Semantic frequency remains long-tailed: over 92% of concepts are linked to fewer than 0.2% of samples after including related concepts.Semantic frequency is computed using CLIP text embeddings, treating concepts within an L2 distance of 4.8 as related.Source: §3.3 · p. 4

4. A Simple “Dirty-Label” Poisoning Attack

The paper first tests a simple dirty-label attack that pairs prompts for a target concept with mismatched images of an unrelated destination concept. Across models, poisoning can redirect or suppress concept generation with hundreds to 1,000 samples, especially for sparse or stylistic concepts.

Experiment Setup

  • 1000 poison samples are highly effective across four text-to-image models, with success measured by both CLIP classification and human inspection.The evaluation covers training from scratch and continuous updating of pretrained models.Source: §4 · pp. 4–5

Attack Notation

  • The attack pairs prompts containing target concept C with images depicting unrelated destination concept A, such as “dog” prompts paired with “cat” images.The mismatched pairs are intended to make the model associate C with A at runtime.Source: §4 · pp. 4–5

Results

  • 500 poison samples achieve 82% human-inspection and 77% CLIP attack success on LD-CC, rising to 98% and 92% with 1000 samples.Source: §4 · p. 5
  • On pretrained SD-V2, SD-XL, and DeepFloyd, 750 poison samples disrupt generation with 85% probability, while 1000 samples push success beyond 90%.These models are harder to poison because each targeted concept averages 986K clean training samples.Source: §4 · p. 5
  • SD-XL shows weak effects at 500 samples but an obvious transformation of outputs at 1000 samples, including attacks on dog, car, cubism, and Sport car.Source: §4 · p. 5, Figure 4 · p. 5
  • The simple attack is more effective against style concepts than object concepts, likely because styles characterize whole images whereas objects occupy specific regions.Source: §4 · p. 5

Concept Sparsity Impact on Attack Efficacy

  • Poisoning succeeds more often for sparser concepts, and semantic frequency predicts attack efficacy better than word frequency.Source: §4 · pp. 5–6

5. Nightshade: an Optimized Prompt-Specific Poisoning Attack

Nightshade is designed to make prompt-specific poisoning both potent and difficult to detect. It aligns poison samples in feature space with a destination concept while keeping them visually consistent with natural images of the targeted concept.

5. Nightshade: an Optimized Prompt-Specific Poisoning Attack

  • Nightshade reduces the poison samples needed for successful attacks by an order of magnitude while avoiding automated and human detection.Source: §5 · p. 6

5.1. Design Goals and Potential Options

  • The attack prioritizes few-sample success because only an unknown fraction of released poison images may enter a trainer’s dataset.Source: §5.1 · p. 6
  • Nightshade must preserve natural text-image consistency because dirty-label mismatches are detectable, while prior perturbation methods such as Glaze required about 800 poison samples.Source: §5.1 · p. 6

5.2. Intuitions and Optimization Techniques

  • Poison potency is increased by selecting focused target prompts, prototypical destination images, and aligned poison samples that reduce variance and conflicting updates.The method aims to overcome noisy, heterogeneous benign training updates.Source: §5.2 · p. 6
  • Nightshade replaces visibly mismatched destination images with perturbed natural target images whose feature representations resemble generated destination anchors.This guided perturbation preserves visual appearance while retaining the intended poisoning direction.Source: §5.2 · pp. 6–7, Figure 5 · p. 7
  • The optimization minimizes feature-space distance between a perturbed target image and an anchor image under a perceptual perturbation budget.F is the model’s image feature extractor, D is feature-space distance, δ is the perturbation, and p is its budget.Source: §5.2 · pp. 6–7

5.3. Detailed Attack Design

  • The detailed algorithm samples high-activation target prompts, generates destination anchors, and optimizes each natural target image into a poison image.It uses a 5K top-ranked prompt pool, random selection of Np prompts, an open-source generator, and LPIPS-bounded perturbations.Source: §5.3 · p. 7
  • The algorithm outputs Np poison text-image pairs from natural aligned data for target concept C, unrelated destination concept A, model M, text encoder Mtext, and budget p.The optimization is solved with an LPIPS constraint and penalty method before adding each pair to the poison dataset.Source: §5.3 · p. 7

6. Evaluation

The evaluation tests Nightshade across training settings, models, concepts, clean-data conditions, composable attacks, and transfer scenarios. It finds strong attacks with few poisons, bleed-through to related prompts, aggregate model degradation, and high effectiveness under transfer and complex prompting.

6.2. Attack Effectiveness

  • ≈100 poison samples successfully attack all four diffusion models, while 50 samples produce 70–80% attack success and 200 samples exceed 84%.Nightshade requires less than 20% of the poison data needed by the simple dirty-label attack.Source: §6.2 · pp. 8–9
  • At 25 and 50 poison samples, usability of images classified as correct falls to 40% and 20%, respectively.Even occasional concept-correct outputs can be incoherent, such as six-legged dogs or malformed cars.Source: §6.2 · p. 9

6.3. Impact of Clean Training Data

  • Poisoning remains effective against clean-data updates: a model poisoned with 500 samples retains 84% attack success after 200K additional clean samples.The poison count needed for over 90% CLIP attack success increases linearly with related clean training data, while successful attacks average 2% of that clean data.Source: §6.3 · pp. 9–10, Table 3 · p. 10

6.4. Bleed-through to Other Concepts

  • Bleed-through corrupts nearby and semantically related prompts, with 69% CLIP attack success at 100 poisons and 88% at 300 for concepts 3.0 < D ≤ 6.0.Related phrases such as “a painting by Michael Whelan” are affected, whereas unrelated prompts such as Van Gogh style show limited impact.Source: §6.4 · p. 10, Figure 14 · p. 10, Figure 15 · p. 11

6.5. Composability Attacks

  • Independent attacks compose: poisoning “dog” to “cat” and “fantasy art” to “impressionism” yields a cat in an impressionism-like style when both concepts are prompted.With 250 concepts poisoned using 100 samples each, alignment falls below 0.24 and FID exceeds 39.6; 500–1000 poisoned concepts produce apparent random noise on untargeted prompts.Source: §6.5 · pp. 10–11, Figure 16 · p. 11, Table 4 · p. 11

6.6. Attack Generalizability

  • Nightshade transfers across model architectures with over 72% CLIP attack success using 200 poison samples and exceeds 89% success across four complex prompt types.Transfer effectiveness drops relative to matched models, while complex prompts and even extremely long prompts remain vulnerable.Source: §6.6 · p. 12, Table 5 · p. 12, Table 6 · p. 12

7. Potential Defenses

The paper evaluates filtering, frequency analysis, alignment filtering, caption regeneration, and gradient outlier detection as defenses against Nightshade. These defenses generally detect or remove too little poison data, while frequency analysis depends on a stable concept distribution.

7. Potential Defenses

  • Filtering high-loss data achieves 73% precision and 47% recall, reducing attack success by less than 5% while risking removal of useful benign outliers.The remaining 159 poison samples are still sufficient for attack success.Source: §7 · p. 12
  • Frequency analysis is constrained because real-world concept distributions vary with time, location, and other collection factors.Even under a fixed distribution, sampling differences produce over 30% frequency changes for more than 19.2% of concepts.Source: §7 · pp. 12–13
  • Alignment filtering achieves only 63% precision and 47% recall for Nightshade, and removing detected samples decreases CLIP attack success by less than 4%.The optimized perturbations affect text-to-image features more than alignment-model features because the extractors serve different objectives.Source: §7 · p. 13
  • Automated captioning has limited defensive effect because generated captions often retain the poisoned or related concepts, reducing attack success by less than 6%.The evaluation uses BLIP-generated captions paired with the original images for retraining.Source: §7 · p. 13
  • Gradient outlier detection identifies fewer than 32% of poison samples at a 10% false-positive rate.The paper notes that robust captioning or customized alignment models could improve defenses, but leaves those approaches for future work.Source: §7 · p. 13

8. Poison Attacks as Copyright Protection

The paper frames Nightshade as a possible countermeasure for content owners facing unauthorized scraping when opt-out and do-not-crawl directives are optional and difficult to verify. Its potency and proactive effect could create incentives for licensing negotiations.

8. Poison Attacks as Copyright Protection

  • Opt-out directives offer limited protection because compliance is optional and violations cannot be reliably detected or verified.The paper describes this as a power asymmetry between model builders and content owners.Source: §8 · p. 13
  • Nightshade could deter unauthorized training because content owners can apply prompt-specific poisoning even when only a small portion of uploaded samples is scraped.The paper presents coordination across content owners and shared poison concepts as a possible deployment model.Source: §8 · p. 13
  • Nightshade may function as a proactive “do-not-train” filter, while post-training unlearning remains difficult to scale and may leave regression to an older model as an alternative.Any effective detection or filtering approach would need to operate over millions or billions of samples.Source: §8 · pp. 13–14

9. Conclusion

The paper demonstrates the practical feasibility of prompt-specific poisoning attacks against text-to-image generative models. It presents these results as evidence of fundamental model limitations and as a possible incentive for licensed training-data procurement.

9. Conclusion

  • Nightshade demonstrates the design and practical feasibility of prompt-specific poison attacks on text-to-image generative models.The conclusion presents the work as an initial step toward understanding these vulnerabilities.Source: §9 · p. 14
  • The authors suggest Nightshade and future attacks may encourage model trainers and content owners to negotiate licensed procurement of training data.This is framed as a potential value rather than an established outcome.Source: §9 · p. 14

A.1. Experiment Setup

The experiments evaluate poisoning across continuous-training and training-from-scratch settings using multiple diffusion architectures, human and CLIP-based assessments, and standard generation metrics.

A.1. Experiment Setup

  • The study tests pretrained SD-V2, SD-XL, and DeepFloyd under continuous training, alongside LD-CC in a training-from-scratch setting.The continuous-training models are updated with 50K randomly selected LAION-5B text/image pairs plus poison data.Source: §A.1 · p. 16
  • The evaluation combines an IRB-approved user study with CLIP alignment and FID measurements on 20K MSCOCO prompts and generated images.The user study includes 185 qualified English-speaking participants, while FID compares generated images with clean MSCOCO images.Source: §A.1 · p. 16

A.2. PCA Visualization of Concept Sparsity

A PCA visualization of text-embedding semantic frequencies shows that common concepts occupy a sparse distribution in the training-data representation.

A.2. PCA Visualization of Concept Sparsity

  • Semantic frequency is sparse across the text-embedding space, with darker points and larger labels indicating higher-frequency concepts.The visualization covers common noun concepts and reports a maximum semantic frequency of 4.17%.Source: §A.2 · p. 16, Figure 18 · p. 17

A.3. Additional Results of Simple Dirty-Label Poisoning Attacks

Simple dirty-label poisoning can substantially disrupt concept generation, including in models with much larger pretrained concept exposure, while styles are more vulnerable than objects and sparse concepts are easier to attack.

A.3. Additional Results of Simple Dirty-Label Poisoning Attacks

  • Injecting 1,000 poisons into LD-CC raises attack success to 98% by human inspection and 92% by CLIP classification.With 500 poisons, success already reaches 82% by human inspection and 77% by CLIP classification, against concepts averaging 2,260 clean samples.Source: §A.3 · p. 17
  • Injecting 1,000 poisons into continuously trained SD-V2, SD-XL, and DeepFloyd pushes attack success beyond 90%.These pretrained models average 986K clean samples per concept, making their attacks more challenging than LD-CC.Source: §A.3 · p. 17
  • Simple poisoning is more effective against style concepts than object concepts because styles are conveyed across entire images rather than localized regions.The comparison is reported for LD-CC and the same pattern is observed on SD-V2, SD-XL, and DeepFloyd.Source: §A.3 · p. 17
  • Attack success increases as concepts become sparser, with semantic frequency correlating more strongly with efficacy than word frequency.The reported trend holds for LD-CC and is also observed on SD-V2, SD-XL, and DeepFloyd.Source: §A.3 · p. 17

A.4. Additional Results

Additional results show that poisoning transfers to related concepts, can be broadened with semantic sampling, remains effective on complex prompts, and is evaluated alongside broader model-performance and review evidence.

A.4. Additional Results

  • With 200 poisons applied to an art style, Nightshade exceeds 91% CLIP attack success on associated artist names without naming the poisoned style.The bleed-through evaluation covers 195 artists associated with 28 styles across four models.Source: §A.4 · p. 17
  • Additional evidence includes attack-success-rate plots across models and evaluators, plus analyses linking poisoning effectiveness to concept sparsity, including correlations and greater effectiveness against style than object concepts.Source: Figure 20 · p. 18, Figure 19 · p. 18, Figure 21 · p. 18
  • Semantic-space sampling increases poison bleed-through by 11% in some cases while reducing performance on the poisoned concept by less than 1%.The enhanced attack selects prompts with high activation similarity to the poisoned concept rather than requiring exact word matches.Source: §A.4 · p. 18
  • Poisoning remains successful on complex prompts containing more than 30 words, exceeding 80% CLIP attack success.The prompts are adapted from a Midjourney showcase by replacing the target concept.Source: §A.4 · p. 18

B.3. Reasons for Acceptance

  • The meta-review characterizes Nightshade as an impactful vulnerability demonstration and an efficient attack requiring fewer than 100 poisoned samples.The review cites comprehensive evaluation and relevance to intellectual-property protection as reasons for acceptance.Source: §B.1 · p. 19, §B.2 · p. 19, §B.3 · p. 19
Loading 2310.13828v3…