Poisoning Web-Scale Training Datasets is Practical

Problem and scope

• The study asks whether attackers can actually poison datasets, focusing on the trust assumptions behind web-content indexes and periodic snapshots.Manual curation is infeasible at billions of examples. Monetary, privacy, and legal restrictions often prevent static dataset distribution, letting attackers identify what content to poison and sometimes when.Source: §1 · p. 1

Split-view poisoning

• Split-view poisoning exploits missing cryptographic integrity protections to serve training clients different content from what dataset curators originally collected.An indexed web page can change between the maintainer’s collection and a client’s later download.Source: §1 · p. 1

Frontrunning poisoning

• Frontrunning poisoning inserts malicious edits immediately before predictable snapshots, exploiting moderation delays and snapshot immutability.Even if moderators subsequently revert the edits, the malicious content persists in the training snapshot.Source: §1 · p. 1

Practical feasibility

• $60 USD could have poisoned 0.01% of LAION-400M or COYO-700M in 2023.The study explores both attacks on 10 popular web-scale datasets, finding them practical even for low-resourced attackers.Source: §1 · p. 2

Defenses and limitations

• Cryptographic integrity checks counter split-view poisoning, while randomized snapshot order or delayed inclusion with trusted-moderator reversions counter frontrunning.Distributing content hashes ensures clients receive the originally indexed and annotated data. However, integrity checks can also prevent benign image re-encoding, resizing, or cropping.Source: §1 · p. 2

Responsible disclosure

• Six of the 10 studied datasets adopted the recommended integrity-check implementation following responsible disclosure.The authors notified all affected dataset maintainers, supplied integrity-check patches for the most popular web-scale dataset downloader, and notified Wikipedia about frontrunning.Source: §1 · p. 2

Towards uncurated datasets

• 11 trillion training tokens would be required for a compute-optimal 500-billion-parameter model under Chinchilla scaling laws [36].This exceeds the training data used for models of this size by over 10× [23]. Expensive curation and growing data requirements have encouraged scraping increasingly broad, untrusted web sources.Source: §2 · p. 2

Security risk of poisoning attacks

• 0.001% poisoning rates have proved effective for certain attacks on uncurated datasets [17], [18], including targeted misclassification and backdoors [22],.Early attacks targeted fully supervised classifiers trained on curated datasets [22],, [88]. Unlike attacks seeking alterations indiscernible to human annotators, attacks on uncurated datasets need not satisfy that stealth requirement.Source: §2 · p. 2
• The practical gap is how attackers can place poisoned samples in training datasets without predicting which parts of the web will be collected.The paper addresses this placement problem rather than merely establishing that poisoning a fraction of a web-scale dataset could cause harm.Source: §2 · p. 2

Auxiliary risks related to data quality

• Uncurated datasets raise fairness, bias, and ethical risks [13], [74],, while filtering can erase marginalized experiences [12].Birhane et al. [12] identify harmful content in LAION-400M, and language datasets retain hate speech and sexually explicit content even after filtering. Filtering choices should consider both security implications and broader data quality.Source: §2 · p. 2

3.1. Terminology

• Distributed datasets publish URLs and auxiliary data without content snapshots; centralized datasets publish periodically refreshed snapshots of content collected from URLs.Clients obtain distributed datasets by crawling the referenced URLs themselves.Source: §3.1 · pp. 2–3

3.2. Threat Model

• A low-resource adversary tampers with a small number of existing URLs so subsequent training produces poisoned results for a desired input.The attacker knows only the published URLs, lacks insider knowledge, cannot change auxiliary data or URL membership, and receives no assistance from honest maintainers, curators, or downloaders. Network interference and website exploitation are outside scope.Source: §3.2 · p. 3, §3.3 · p. 3
• The attacks assume distributed-dataset clients omit cryptographic integrity comparisons, while centralized curators need time to detect malicious edits.For snapshot time t_i and detection delay Δ, edits made within t_i − Δ ≤ t̂_i ≤ t_i can evade detection before collection. Cryptographic integrity checks and randomized crawling can mitigate the attacks.Source: §3.2 · p. 3

3.3. Attack Scenarios

• Split-view poisoning uses sustained control of indexed web resources, including purchased expired domains, to replace benign content before clients download it.Attackers can prioritize domains serving multiple dataset URLs to minimize cost, or target domains associated with desired auxiliary data. Dataset indexes often remain unchanged as content goes stale, and no modern datasets include downloaded-content integrity checks.Source: §3.3 · p. 3
• Frontrunning poisoning inserts malicious content immediately before a predictable dataset snapshot, allowing temporary edits to enter the dataset before moderators revert them.Wikipedia’s documented linear snapshot sequence [4] lets attackers predict article collection down to the minute. Randomly timed vandalism is unlikely to affect collection because edits are reverted within minutes on average.Source: §3.3 · p. 4

4.1. Our Attack: Purchasing Expired Domains

• Expired-domain purchases let attackers replace indexed images while captions remain fixed, exploiting temporary domain ownership and residual trust in distributed datasets.The attack applies to distributed datasets generally, although the evaluation focuses on image-text datasets. Lax domain quality assurance permits acquisition of domains that have already been expired for months.Source: §4.1 · p. 4, §4.2 · p. 4

4.3.1. Results.

• Over 800 downloads could have been poisoned during six months, with even the oldest, least-accessed datasets receiving at least three downloads monthly.Only downloads after domain takeover are vulnerable; newer datasets offer fewer purchasable images but more potential victims. Billion-image datasets are downloaded whole less frequently because they supply subsets such as PMD [92] and LAION-Aesthetics.Source: Table 1 · p. 5, §4.3 · p. 5, §4.3.1 · p. 6

4.2. Quantifying the Attack Surface

• 0.1% of COYO-700M images were already hosted on expired domains purchasable for under $1,000 on release day.LAION-5B's URL/caption hashes do not protect image contents, while COYO-700M's pHash is not adversarially robust [41]. FaceScrub and PubFig provide cryptographic hashes but no official verifying downloader, and nearly all third-party downloaders ignore those hashes.Source: §4.2 · p. 5

4.3. Measuring the Attack Impact

• Six purchased domains per dataset enabled request monitoring, using the three most image-rich available domains and three randomly selected available domains.Monitoring ran for twelve months beginning August 2022. Detection required one IP to request at least 90% of controlled dataset URLs across all six domains, with at least 50% request precision; the single-IP assumption can underestimate downloads.Source: §4.3 · pp. 5–6

4.2. Quantifying the Attack Surface

• 0.01% of each studied dataset could be controlled for less than $60 USD per year through expired-domain purchases.A $1,000 budget purchased control over 0.02%–0.79% of images across the ten datasets, prioritizing domains by images per dollar. Older datasets contained more expired domains and therefore offered larger purchasable fractions.Source: §4.2 · pp. 4–5, Figure 1 · p. 6

4.3.1. Results.

• Conceptual 12M downloads mostly followed index order, supporting downloader identification, but ranged from hours to weeks and included parallel chunks and resumed transfers.A user agent hardcoded in img2dataset accounted for 77% of traffic, suggesting that tool generated most requests. The researchers never poisoned public downloads: owned image URLs returned 404 responses, and domain landing pages explained the study and offered domain restoration.Source: §4.3.1 · pp. 6–7, Figure 2 · p. 7

4.4. Is This Attack Exploited in the Wild?

• Checking image changes and domain ownership found no evidence of malicious split-view poisoning in CC3M or LAION-400M.CC3M's candidates were one advertising squatter and two unchanged owners whose images had been re-encoded. LAION-400M showed 175M and 183M modified images in successive snapshots but no attack signature; reliance on perceptual comparisons could miss adversarially preserved similarity.Source: §4.4 · p. 7

4.5. Putting It All Together

• 1,000 poisoned images (0.00025% of the dataset) achieved 60% targeted zero-shot misclassification success and above 90% success against the Stable Diffusion Safety Filter [78].The LAION-400M experiments modified images without controlling captions, selecting purchasable domains costing $1,000 in total. Poisoned images were substituted locally to simulate attacks without harming other downloaders.Source: §4.5 · pp. 7–8

5.1. Our Attack: Editing Wikipedia

• Frontrunning inserts malicious edits immediately before Wikipedia snapshots capture them, preserving poisoned content even when the live edits are quickly reverted.Wikipedia-derived training datasets use official snapshots rather than crawling the live encyclopedia. The attack therefore replaces sustained control of content with the ability to predict its download time precisely.Source: §5.1 · p. 8, §5 · p. 8

5.2. Predicting Checkpoint Times

• Predictable rolling snapshots let attackers estimate each article’s next scrape time from its previous offset and the publicly reported snapshot start.English Wikipedia snapshots begin on the 1st and 20th monthly, with parallel workers scraping sequential article-ID chunks. Stable crawling rates support t_i ≈ t_0 + (t_i,prev − t_0,prev), but attackers must predict individual article times rather than merely the dump’s start.Source: §5.2.2 · p. 9, §5.2.1 · p. 9

5.2.3. Determining the Article Snapshot Time.

• Sequential article processing tightens scrape-time bounds from included and excluded edits, enabling per-worker linear fits of previous snapshot times.An article’s last included and first excluded edits initially bound its scrape time, often several days apart. Included edits on earlier articles and excluded edits on later articles tighten those bounds before fitting.Source: §5.2.3 · p. 9

5.2.4. Evaluating our Predictions.

• 27 minutes bounds the average previous-snapshot estimation error; next-snapshot predictions usually fall within roughly 30 minutes of retrospective estimates.Exact scrape times are unknown, so evaluation uses reconstructed intervals and retrospective fits. Snapshots speed up over time, causing the extrapolation procedure to typically overestimate the next scrape time.Source: Figure 4 · p. 9, §5.2.4 · pp. 9–10, Figure 5 · p. 10

5.3. Estimating Revision Speed

• Roughly 35% of English Wikipedia reversions take over 30 minutes, while combined snapshot-timing uncertainty averages about one hour.The authors estimate reversion speeds passively using reversion-comment strings, treating the immediately preceding edit as the reverted edit. This assumption can understate how long malicious content survives.Source: Figure 6 · p. 10, §5.3 · p. 10, §5.4 · p. 10

5.4. Putting It All Together

• 6.5% of Wikipedia documents could be poisoned under the authors’ conservative timing analysis, absent additional defensive measures.An optimized timing adjustment balances missing the snapshot against having edits reverted beforehand. Rate limits and IP bans could reduce success; the estimate optimizes this adjustment retrospectively but assumes only one attempt per article.Source: §5.4 · p. 11

5.5. Multilingual Wikipedia

• 22 (63%) of 35 analyzed non-English Wikipedias were easier to poison than English, with feasible poisoning rates ranging from 0.95% to 25.3%, with median 8.2%.Smaller, more stable snapshots improve predictability, and multilingual datasets often depend more heavily on Wikipedia. Incomplete reversion detection and conservative intervals limit these estimates; sparse edits can depress bounds, while IP bans and rate limits make such large rates unlikely.Source: Figure 7 · p. 11, §5.5 · p. 11

5.6. Ethical Considerations

• The study made no Wikipedia edits and disclosed its attack analysis and defense recommendations to Wikimedia researchers, who acknowledged the vulnerability before publication.The authors interacted only by downloading official datasets, accepting analytical limitations to avoid harming Wikipedia’s editor community.Source: §5.6 · p. 11

6.1. Existing Trust Assumptions

• The proposed defenses assume honest maintainers, curators, and downloaders, including unpoisoned dataset indexes and faithfully performed integrity checks.Maintainers must distribute identical indexes, while curators must distribute identical corpora and control snapshot timing. These assumptions preserve existing trust relationships rather than defending against compromised ecosystem participants.Source: §6.1 · p. 12

6.2. Preventing Split-View Poisoning

• Cryptographic hashes prevent split-view poisoning by letting downloaders reject content that differs from a trusted, pre-attack copy.Maintainers or trusted third parties attach hashes of original raw content to the dataset index; downloaders verify retrieved content using a cryptographic function such as SHA-256.Source: §6.2 · p. 12
• CC3M, CC12M, LAION-2B-en, LAION-2b-multi, LAION-1B-nolang, and LAION-400M adopted SHA-256 hashes, while the authors added optional verification to img2dataset.Existing hash publication alone was insufficient: among six popular downloaders examined per dataset, only one each for PubFig and FaceScrub implemented verification. FaceScrub’s checked by default; PubFig’s required a separate verification script.Source: §6.2 · p. 12
• 1.1 million of 2.9 million still-hosted Conceptual Captions images matched their original hashes, illustrating the utility cost of strict integrity checks.Comparing 2018 originals with 2023 downloads, 1.8 million hosted images had changed. Perceptual hashes could preserve more benign changes, but adversarially modified poisoned images could evade them, [41], [95].Source: §6.2 · p. 12

6.3. Preventing Frontrunning Poisoning

• 89.5% of English Wikipedia articles would be protected from random malicious modification by uniformly randomizing snapshotting over 24 hours, assuming no automated Sybil reinsertion.Randomized snapshot order and longer collection periods increase the control duration attackers need relative to moderation latency, averaging 2.5 hours here. Protection weakens if attackers automatically restore reverted edits using Sybil accounts.Source: §6.3 · pp. 12–13
• 90% versus 50% reversion rates for one-day versus five-minute review windows imply a fivefold reduction in English Wikipedia vandalism.Curators could hold snapshots beyond the detection interval and incorporate only trusted moderators’ corrections before release. Protection remains incomplete because detection latency varies across articles and languages, and human review may miss machine-targeted changes.Source: §6.3 · p. 13

6.4. Preventing Poisoning in General

• General web-scale poisoning defenses lack trusted reference snapshots, reliable moderation, and principled domain-trust rules.Cross-site consensus could require attackers to poison more websites, but inherits consensus challenges seen in blockchains. Application-specific defenses require knowledge of downstream content consumption, vectorization, and deconfliction, and remain future work.Source: §6.4 · p. 13

6.5. Transparency to Improve Trust

• The authors propose data transparency for consistent dataset distribution and binary transparency against selective inclusion of malicious downloader modules [2].Data transparency, akin to certificate transparency, could expose inconsistent indexes and help detect inaccurate annotations or expired URLs; curators could similarly ensure identical corpora. These mechanisms would support continuously updated datasets with multiple maintainers and curators.Source: §6.5 · p. 13

7. Conclusion

• 0.01% of examples can suffice to poison a model, even when attackers can target only a fraction of a curated dataset.The demonstrated attacks are low-cost, extremely practical, and feasible today.Source: §7 · p. 13
• Dataset publishers and maintainers should consider integrity checks, randomized or time-gated snapshots, or application-specific defenses.These are defenses to consider, rather than a claim that any single defense eliminates poisoning risk.Source: §7 · p. 13
• Machine learning researchers should reassess trust assumptions in web-scale data and explore solutions that do not assume a single root of trust.Source: §7 · p. 13
• Future research should examine raw-content-only attackers, practical attack costs, and the efficacy of more permissive near-duplicate integrity checks.The proposed threat models allow edits to raw content but not auxiliary data such as labels; more permissive integrity checks may themselves be vulnerable.Source: §7 · p. 13

Appendix A. Further Discussion for Text Datasets

• The discussion covers targeted poisoning attacks on Wikipedia before examining vulnerabilities in the Common Crawl dataset.Source: Appendix A · p. 17

A.1. Annotating Reversions

• Language-specific reversion vocabularies are identified manually, starting from comments tagged with the English word “reversion” or abbreviation “rv” and finding terms meaning “revert,” “undo,” or similar.These vocabularies emerge from each Wikipedia contributor community rather than a fixed list.Source: §A.1 · p. 17
• Comments containing each newly identified term are sampled to verify that it uniquely tags reversions, captures additional reversion instances, and does not produce too many false positives.Candidate terms must also appear in comments already identified as automated or manual reversions.Source: §A.1 · p. 17
• The authors consider their imperfect reversion annotations sufficient to support findings that frontrunning attacks remain possible and may be more powerful on non-English Wikipedias.No author is an active contributor to any language’s Wikipedia, so they do not expect any language’s list to be perfect.Source: §A.1 · p. 17

Shared attack mechanism

• Both attacks bring fixed image embeddings closer to target textual-label embeddings, with all attacks run in parallel to minimize expensive CLIP retraining.Parallel execution is the experiment’s key technical constraint.Source: Appendix B · p. 18

Object-misclassification objective

• 10 target labels each receive 1000 image replacements paired with label-containing captions, aiming to make CLIP’s ImageNet zero-shot classifier assign incorrect labels.Each label’s replacements use one unrelated image, with domains spanning all selected pairs costing at most $1,000 USD collectively. The classifier selects the maximum-cosine-similarity label among ImageNet’s 1000 classes.Source: Appendix B · p. 18

NSFW objective

• 10 benign images each replace 1000 images paired with captions marked UNSAFE in LAION 400M metadata, aiming to induce false NSFW labels.The targeted Stable Diffusion 1.4 filter in Hugging Face diffusers thresholds cosine similarity between CLIP image embeddings and textual NSFW concepts [78]. Selected domains cost less than $1,000 USD collectively.Source: Appendix B · p. 18

The experiment

• 60% of object-misclassification targets and 90% of NSFW-targeted images were successfully attacked when both methods were trained simultaneously on LAION 400M.The experiment trained an OpenCLIP ViT-B-32 model for 32 epochs with a batch size of 3072 on 16 A100 GPUs.Source: Appendix B · p. 18

Appendix C. Landing Page for Purchased Domains

• The researchers purchased expired domains in August 2022 to measure queries to domains that previously hosted dataset images.The landing page disclosed that the purchases supported research into how machine-learning datasets change over time.Source: Appendix C · p. 18
• Purchased domains returned 404 errors for every request except the homepage, requiring no additional action from dataset downloaders.Without the purchases, the domains would have returned NXDOMAIN rather than content. The researchers tested that standard image-downloading tools skipped their 404 responses.Source: Appendix C · p. 18
• The landing page disclosed possible temporary request-metadata logging, promised deletion after the study, and offered deletion upon a participant’s request.Logging was intended to measure the prevalence of scraping the purchased domains.Source: Appendix C · p. 18
• The researchers offered to return purchased domains to their original owners upon request and otherwise planned to let them expire after the study.Source: Appendix C · p. 18

Appendix D. Meta-Review

• The 2024 IEEE Symposium on Security and Privacy program committee prepared the meta-review evaluating the paper’s practical dataset-poisoning attacks.Source: Appendix D · p. 19, §D.1 · p. 19

D.1. Summary

• The meta-review concludes that substantial fractions of web-scale datasets can be poisoned at reasonable attacker cost and notes proposed low-overhead defenses.Split-view poisoning exploits differences between provider and consumer views, primarily through expired-domain content changes. Frontrunning temporarily poisons crowd-sourced content so that snapshots capture malicious examples.Source: §D.1 · p. 19

D.2. Scientific Contributions

• The scientific-contributions assessment characterizes the paper as a valuable step forward in an established field.Source: §D.2 · p. 19

• The paper’s main contribution is demonstrating that poisoning web-scale training datasets is practical.Source: §D.3 · p. 19
• Concrete examples show how web-scale training datasets could be poisoned.Source: §D.3 · p. 19
• The measurement aspects of the work were also considered valuable.Source: §D.3 · p. 19

D.4. Noteworthy Concerns

• The meta-review argues that attacks requiring absent integrity protections neither demonstrate failures of established defenses nor reveal previously overlooked vulnerabilities.It therefore questions whether the paper meets either expectation for an attack paper.Source: §D.4 · p. 19
• The meta-review requests more discussion of scalable defenses, distinguishing trustworthy from untrustworthy edits, and whether poisoning remains feasible at scale.Source: §D.4 · p. 19

Appendix E. Response to the Meta-Review

• The authors argue that missing integrity protections make the attacks relevant, claiming every large-scale dataset released in the preceding decade lacked these protections.They report that three datasets released after the paper appeared on arXiv still omitted hashes.Source: Appendix E · p. 19
• The authors argue that research remains valuable when effective defenses exist but are rarely used, drawing analogies to SQL injection and memory corruption.They identify prepared statements and memory-safe languages as the corresponding defenses in those examples.Source: Appendix E · p. 19
• A factor of three reduction in CC-3M’s size when hashes are added illustrates the utility cost of requiring integrity protection.The authors present this loss of utility as a reason datasets may not mandate cryptographic hashes.Source: Appendix E · p. 19
• LAION-400M’s authors provided optional hashes after notification, which the paper attributes to users preferring model accuracy over security.Source: Appendix E · p. 19