Source-linked AI summary
Poisoning Web-Scale Training Datasets is Practical
Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, Florian Tramèr
TL;DR
Web-scale training datasets are difficult to curate and rely on trust assumptions that may leave them open to poisoning. The paper introduces split-view and frontrunning attacks targeting mutable URL content and predictable snapshots, respectively. These attacks were practical across 10 datasets, including a $60 USD path to poisoning 0.01% of LAION-400M or COYO-700M, prompting integrity and timing-based defenses.
Problem
The paper asks whether attackers can actually place poisoned samples into real-world web-scale training datasets, beyond prior work that assumed dataset modification.
Method
The paper develops split-view poisoning for distributed URL indexes and frontrunning poisoning for periodically snapshotted crowd-sourced content.
Results
The attacks were practical on 10 popular datasets, including poisoning 0.01% of LAION-400M or COYO-700M for $60 USD in 2023.
Takeaways & Limitations
The authors recommend integrity checks and randomized or time-gated snapshots, and disclosed the findings to affected dataset maintainers.
Takeaways & Limitations
The analysis includes practical constraints such as edit rate limiting and IP bans, while the study itself remained passive to avoid harming the Wikipedia editor community.
Abstract
from arXiv · showhide
Deep learning models are often trained on distributed, web-scale datasets crawled from the internet. In this paper, we introduce two new dataset poisoning attacks that intentionally introduce malicious examples to a model's performance. Our attacks are immediately practical and could, today, poison 10 popular datasets. Our first attack, split-view poisoning, exploits the mutable nature of internet content to ensure a dataset annotator's initial view of the dataset differs from the view downloaded by subsequent clients. By exploiting specific invalid trust assumptions, we show how we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for just $60 USD. Our second attack, frontrunning poisoning, targets web-scale datasets that periodically snapshot crowd-sourced content -- such as Wikipedia -- where an attacker only needs a time-limited window to inject malicious examples. In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses.
1. Introduction
Web-scale datasets make exhaustive curation infeasible, while existing trust assumptions leave their mutable contents vulnerable to practical poisoning. The paper introduces split-view and frontrunning attacks, demonstrates low-cost feasibility, and proposes integrity and timing-based defenses.
- Motivation: Web-scale datasets have made manual quality assurance infeasible, despite neural networks’ resilience to label noise.The paper frames this as a quantity-over-quality tradeoff in internet-crawled training data.
- Research question: The paper asks whether an adversary can actually place poisoned samples into a real-world web-scale dataset.It focuses on this feasibility question rather than the already-studied impact, stealth, and defense questions.
- Attacks: Two attacks exploit mutable web content: split-view poisoning targets distributed URL indexes, while frontrunning poisoning targets predictable snapshots of crowd-sourced content.The attacks rely on differences between curator and client views, or on timing malicious edits immediately before collection.
- Results: $60 USD could have poisoned 0.01% of LAION-400M or COYO-700M in 2023, demonstrating practical feasibility for a low-resourced attacker.The feasibility study covered 10 popular web-scale datasets.
- Defenses: Integrity verification prevents split-view poisoning, while randomized or delayed snapshots and trusted reversions address frontrunning poisoning.These defenses aim to ensure consistent content views and reduce attackers’ ability to exploit predictable collection timing.
- Limitations: The proposed integrity checks can reject benign image changes such as re-encoding, resizing, or cropping.The paper discusses this compatibility tradeoff alongside lower-trust future defenses.
- Responsible disclosure: Six of the 10 dataset maintainers followed the recommended integrity-check implementation after responsible disclosure.The authors also patched a popular downloader and notified Wikipedia about its frontrunning vulnerability.
2. Background & Related Work
Web-scale training data is increasingly necessary for large models but remains difficult to curate, creating both security and broader data-quality risks. Prior work established poisoning impacts, while this paper addresses how attackers could place poisoned samples into datasets in practice.
- Uncurated datasets: Large models require enormous training datasets, but curating web-scale data is expensive and available training data is becoming a limiting factor.A compute-optimal 500-billion-parameter model was estimated to require 11 trillion training tokens.
- Security risk: Uncurated datasets are attractive poisoning targets because attackers can alter training data to induce targeted model misclassification or backdoors.Prior work found poisoning rates as low as 0.001% effective for certain attack classes.
- Research gap: The paper focuses on the unresolved feasibility question of how an attacker can place poisoned samples into collected web-scale training data.Earlier work largely assumed dataset modification and studied its consequences, stealth, defenses, or attacks on defenses.
- Data quality: Uncurated data also raises fairness, bias, and ethical concerns beyond security.LAION-400M is cited as containing explicit, racist, stereotyped, and otherwise problematic image-text content.
- Data quality: Curation and filtering can themselves erase marginalized experiences, so security implications must be considered alongside broader data-quality metrics.The passage cautions against evaluating filtering only by what content it removes.
3. Threat Model & Attack Scenarios
The threat model covers honest maintainers and low-resource attackers who exploit two web-scale dataset formats: mutable URL indexes and periodically refreshed snapshots. The attacks depend on weak integrity checking or predictable collection timing, with split-view and frontrunning mechanisms targeting the respective formats.
- Dataset formats: Web-scale datasets are either distributed URL indexes or centralized snapshots that periodically re-collect changing web content.Distributed datasets omit the original resources, while centralized datasets publish collected samples for easier access.
- Dataset formats: Clients download distributed datasets by crawling URLs or centralized datasets by downloading published snapshots, often through third-party tools.The two delivery paths expose different collection points for poisoning.
- Threat model: The attacker is assumed to be low-resource and unskilled, able to tamper with a small number of indexed URLs but unable to alter auxiliary labels or add and remove URLs.Maintainers, curators, and downloaders are assumed honest and provide no insider assistance.
- Threat model: The attacks assume clients do not verify distributed-dataset integrity and that centralized curators need time Δ to detect malicious changes.For centralized data, poisoning can occur during the interval ti − Δ ≤ t̂i ≤ ti before snapshot inclusion.
- Scope: The study restricts itself to poisoning mechanisms specific to dataset collection and excludes network interference or website exploitation.Those excluded vulnerabilities would only improve attack success rates.
- Split-view poisoning: Split-view poisoning exploits mutable URL content, including expired domains, so clients receive content different from what maintainers originally indexed.The attacker can prioritize cheap domains hosting many indexed URLs or content with selected auxiliary data.
- Frontrunning poisoning: Frontrunning poisoning temporarily edits crowd-sourced content immediately before predictable snapshots, allowing malicious data to persist despite later reversion.Wikipedia is a key setting because vandalism is usually reverted within minutes while snapshot access can be predictable.
4. Split-View Data Poisoning
Split-view poisoning exploits expired domains and mutable web content so later dataset downloaders receive attacker-controlled images instead of the curator’s original data. Across ten datasets, the attack surface is practically reachable, regularly accessed, and large enough to meet prior poisoning thresholds, although older downloads may already be cached and no exploitation evidence was found in two examined datasets.
- Attack mechanism: Expired domains let attackers return arbitrary content for indexed images, creating a split view between dataset curation and later client downloads.The attack leverages DNS ownership leases and weak dataset quality assurance rather than modifying the original dataset index.
- Attack surface: 0.02%–0.79% of images in each of ten studied datasets could be controlled with a modest budget, exceeding prior poisoning thresholds.Prior work often requires poisoning only 0.01% of an uncurated dataset.
- Limitations: The measured attack surface has a scope caveat: users who downloaded datasets before domain takeover were likely not vulnerable, although fresh downloads remained frequent.The analysis measured vulnerability in August 2022, while many datasets had been constructed years earlier.
- Attack surface: COYO-700M already had 0.1% of images on expired domains costing under $1,000 USD on its 30 August 2022 release date.The paper attributes this to the nearly year-long collection process, during which early scraped domains could expire before release.
- Downloader exposure: Fresh exposure remained substantial: every dataset had at least 3 downloads per month, yielding over 800 potentially poisonable downloads during six months of tracking.Newer datasets were accessed more often, while older datasets offered a larger fraction of purchasable images.
- Downloader exposure: Download monitoring found mostly linear URL access, variable crawl speeds, parallel chunking, and resumptions across different IP addresses.The linear ordering strongly confirmed that logged requests represented dataset downloads because the index ordering was random.
- Observed exploitation: The study found no evidence that split-view poisoning had already been exploited in CC3M or LAION-400M, despite the attack vector’s long availability.CC3M showed one perceptual-similarity signature attributable to a domain squatter’s advertising, while LAION-400M showed none across three versions.
5. Frontrunning Poisoning
Frontrunning poisoning exploits predictable Wikipedia snapshot timing: attackers can make malicious edits immediately before an article is crawled, leaving them in an otherwise static dump even after reversion. The analysis finds practical poisoning rates, especially for multilingual Wikipedias, while noting passive measurement and operational constraints.
- 5.1. Our Attack: Editing Wikipedia: Wikipedia-derived datasets use static dumps because direct crawling of the live site is forbidden, creating a target for timed edits.An edit made immediately before scraping can persist in the snapshot forever, even if quickly reverted afterward.
- 5.2. Predicting Snapshot Times: The attack predicts article crawl times by exploiting known snapshot starts and sequential, nearly linear processing within parallel jobs.Tracking included and excluded edits narrows each article’s snapshot interval; linear fitting then extrapolates the next snapshot time.
- 5.2.3–5.2.4. Evaluating Predictions: 27 minutes bounds the average furthest-boundary estimation error, while most next-snapshot predictions fall within roughly 30 minutes of ground truth.Predictions are biased early because snapshots slightly speed up over time, requiring later adjustment.
- 5.3. Estimating Revision Speed: Combining prediction and snapshot-time uncertainty with revision-speed estimates lets an attacker time edits within approximately one hour of an article’s snapshot.The revision analysis uses 18 months of Wikipedia edits and conservatively treats the immediately preceding edit as the reverted one.
- 5.4. Putting It All Together: 6.5% of Wikipedia documents could be poisoned under the conservative analysis without defensive measures.The estimate assumes a single attempt per article and excludes factors such as edit rate limiting and IP bans that could reduce success.
- 5.5. Multilingual Wikipedia: 22 (63%) of 35 non-English Wikipedias analyzed were easier to poison than English Wikipedia, with feasible rates from 0.95% to 25.3% and a median of 8.2%.The authors attribute greater vulnerability to smaller, shorter checkpoint procedures and more stable checkpointing, while multilingual datasets may rely more heavily on Wikipedia.
6. Defenses
The paper proposes integrity checks for split-view poisoning and timing-based defenses for frontrunning poisoning, while documenting their assumptions, implementation status, and remaining limitations. It also identifies broader web-scale datasets where existing defenses are harder to apply.
- Split-view poisoning: Integrity checks compare downloaded content with maintainer-provided cryptographic hashes and discard mismatches, preventing split-view attacks when content remains unchanged.The proposed check uses hashes such as SHA-256.
- Split-view poisoning: Six datasets adopted recommended SHA-256 integrity checks, and img2dataset added verification support as part of responsible disclosure.The affected datasets include CC3M, CC12M, LAION-2B-en, LAION-2b-multi, LAION-1B-nolang, and LAION-400M.
- Split-view poisoning: Integrity checks can reduce dataset utility when benign content changes, because changed files no longer match their original hashes.For Conceptual Captions 3M, 1.1 million of 2.9 million still-hosted images matched the original hashes.
- Frontrunning poisoning: Timing defenses increase the interval attackers must retain control before snapshot inclusion, allowing detection and reversion to block more attacks.Randomized snapshot order and delayed finalization are proposed approaches.
- Frontrunning poisoning: 89.5% of articles could be protected if English Wikipedia snapshotting were uniformly randomized over 24 hours, given a 2.5-hour average review time.This estimate assumes attackers cannot use Sybil accounts to reintroduce malicious edits.
- Frontrunning poisoning: Frontrunning defenses cannot prevent attacks entirely because detection times vary and trusted curators may be difficult to establish.Attackers may target less active articles or languages with fewer moderators.
- General web-scale datasets: General datasets such as Common Crawl are harder to defend because they lack a trusted snapshot, trusted curator, and reliable version-change signal.The paper leaves application-specific solutions to future work.
- Broader defenses: The paper recommends transparency around distributed dataset indexes and hashes to detect inconsistent distribution, inaccurate metadata, and expired URLs.The proposal is compared with certificate transparency.
7. Conclusion
The paper concludes that web-scale datasets are vulnerable to low-cost, practical poisoning attacks and calls for reassessing their trust assumptions. It presents integrity checks and randomized or time-gated snapshots as defenses while framing the work as a starting point for further research.
- Conclusion: Web-scale datasets are vulnerable to low-cost poisoning attacks that could be carried out today.The conclusion emphasizes that corrupting only a fraction of examples can be sufficient to poison a model.
- Conclusion: Dataset publishers and maintainers should consider integrity checks, randomized snapshots, time-gated snapshots, or application-specific defenses.The paper also calls for solutions that do not assume a single root of trust.
- Conclusion: The paper identifies future research directions including broader threat models, practical attack costs, and more permissive integrity checks.These directions address attacks on auxiliary data and the efficacy of near-duplicate checks.
Appendix A. Further Discussion for Text Datasets
The appendix discusses poisoning risks in text datasets, focusing on targeted attacks against Wikipedia and the Common Crawl dataset.
- Further Discussion for Text Datasets: The discussion covers targeted poisoning attacks on Wikipedia before considering vulnerabilities in Common Crawl.
A.1. Annotating Reversions
The authors annotate Wikipedia reversions by identifying language-specific terms and validating them against sampled reversion comments. They caution that the resulting lists are imperfect but sufficient to examine cross-language trends.
- Annotating Reversions: The authors construct language-specific reversion-word lists from automated and manual reversion comments.The words roughly translate to “revert,” “undo,” or similar terms and may include “rv.”
- Annotating Reversions: They sample comments to verify that the identified terms capture reversions while limiting false positives.
- Annotating Reversions: The lists are imperfect because no author is an active contributor to every language’s Wikipedia, but the analysis supports two reported trends.The trends are that frontrunning remains possible on non-English Wikipedias and may be more powerful there.
Appendix B. LAION Attack Details
The appendix describes two poisoning objectives against CLIP-based systems and reports their success when applied simultaneously during LAION-400M training.
- Attack setup: Both attacks target CLIP embeddings by bringing fixed-image embeddings close to target textual labels.The experiments require parallel attacks because retraining CLIP is expensive.
- Attack objectives: The object-misclassification objective makes CLIP's zero-shot ImageNet classifier assign a chosen image an incorrect target label.The classifier selects the ImageNet label with maximum cosine similarity to the image embedding.
- Attack objectives: The object-misclassification setup selects 10 target classes with at least 1000 matching captions and caps total domain costs at $1,000 USD.For each class, the authors select 1000 caption-image pairs whose captions contain that class label.
- Attack objectives: The NSFW objective makes Stable Diffusion 1.4's filter mislabel a benign image as NSFW using CLIP-space similarity to textual NSFW concepts.The filter is described as a cosine-similarity threshold function.
- Results: 90% of targeted images succeeded in the NSFW attack, while 60% of object-misclassification targets received the chosen incorrect label.The attacks were trained simultaneously for 32 epochs on LAION-400M using OpenCLIP, batch size 3072, and 16 A100 GPUs.
Appendix C. Landing Page for Purchased Domains
The purchased-domain landing page explained the study, described domain behavior and logging, addressed dataset effects and ownership, and stated that publication was expected within several months.
- Landing-page content: The landing page said the domains were purchased to study how machine-learning datasets change over time and how often expired domains are queried.It stated that the domains had hosted dataset images before their previous owners allowed them to expire.
- Landing-page content: Purchased domains returned 404 errors for all requests except the home page, so dataset users would not receive replacement content from them.The page said that without the purchase, the URLs would have returned NXDOMAIN instead.
- Landing-page content: The researchers said they might temporarily log request metadata, would delete it after the study, and would honor deletion requests.Visitors were invited to contact the researchers if they preferred not to participate.
- Landing-page content: The page offered to return domains to their original owners and said the domains would expire after the research study ended.This response addressed former owners asking to reclaim their domains.
- Landing-page content: The researchers reported testing that standard image-download tools skip the 404 responses, so dataset scrapers need not take special steps to avoid the domains.The page also said the study would be published after completion, expected within the next several months.
- Review context: The meta-review described the paper as investigating practical poisoning of real-world web-scale datasets through split-view and frontrunning attacks.It characterized split-view poisoning as exploiting differences between provider and consumer views, including expired-domain content.
D.3. Reasons for Acceptance
The program committee viewed the paper's main contribution as demonstrating that poisoning web-scale training datasets is practical, with valuable measurement work.
- Reasons for Acceptance: The paper's main contribution is showing that poisoning web-scale training datasets is practical through concrete attack examples.The review also regarded the measurement aspects as valuable.
D.4. Noteworthy Concerns
The concerns focus on the attacks' dependence on absent integrity protections, while the authors argue that such protections are uncommon and can reduce dataset utility.
- Noteworthy Concerns: The review criticized the paper for not demonstrating failures of existing defenses or exposing fundamental flaws in established security mechanisms.It judged the attack effective only when standard integrity protection measures are absent.
- Author Response: The authors respond that every large-scale dataset released in the last decade includes the condition of absent standard integrity protection.They also state that three datasets released after the paper's arXiv publication still had the vulnerability.
- Author Response: The authors compare rarely applied dataset defenses with prepared statements and memory-safe languages, arguing that imperfectly deployed defenses remain practically concerning.Their analogy frames research on gaps between available protections and real-world adoption as valuable.
- Author Response: Adding cryptographic hashes to CC-3M reduces its size by a factor of three, and LAION-400M offered hashes optionally because users may prioritize model accuracy.The authors present utility loss and user preference as reasons hashes may not be mandatory.