Source-linked AI summary
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, Tom Goldstein
TL;DR
Machine-learning systems increasingly rely on outsourced and weakly supervised data collection, creating vulnerabilities to poisoning and backdoor manipulation. The paper systematizes these threats, defenses, and threat models into a unified taxonomy, and identifies open problems including clean-label and physically robust backdoor attacks. It also highlights privacy–security tension in federated learning and the lack of controlled comparisons in the literature.
Problem
Automated collection from anonymous and unverified sources leaves training data vulnerable to manipulation that can control or degrade learned-model behavior.
Method
The paper catalogs dataset-creation vulnerabilities, systematizes poisoning and backdoor threat models, reviews defenses, and discusses open problems and their relationships.
Results
The survey develops a unified taxonomy of dataset vulnerabilities and exploits, defenses against them, threat models, and open research problems.
Takeaways & Limitations
The survey identifies clean-label poisoning, physically robust backdoors, and controlled datasets and benchmarks as important directions for improving understanding and comparison.
Takeaways & Limitations
Secure aggregation can prevent screening federated updates for corruptions, illustrating that privacy may conflict with security.
Abstract
from arXiv · showhide
As machine learning systems grow in scale, so do their training data requirements, forcing practitioners to automate and outsource the curation of training data in order to achieve state-of-the-art performance. The absence of trustworthy human supervision over the data collection process exposes organizations to security vulnerabilities; training data can be manipulated to control and degrade the downstream behaviors of learned models. The goal of this work is to systematically categorize and discuss a wide range of dataset vulnerabilities and exploits, approaches for defending against these threats, and an array of open problems in this space. In addition to describing various poisoning and backdoor threat models and the relationships among them, we develop their unified taxonomy.
1 Introduction
The paper surveys how open-world dataset creation exposes machine-learning systems to poisoning and backdoor threats, then organizes attacks, defenses, threat models, and open problems into a unified perspective.
- Threat landscape: Open-world data collection from anonymous and unverified web sources allows outsiders to passively or actively manipulate training datasets.Examples include corrupted web data harvested by bots and corrupted samples submitted directly by privileged outsiders.
- Paper scope: The article catalogs dataset-creation vulnerabilities, reviews their exploitation of machine-learning systems, compares threat models, and identifies open problems for assessing severity and improving defenses.It distinguishes its scope from inference-time evasion attacks and related focused surveys.
- Attack taxonomy: Training-only attacks manipulate training data or labels without requiring access to test-time inputs after deployment.They can target from-scratch training or transfer learning and may differ in their optimization formulation or crafting heuristic.
- Attack taxonomy: Backdoor attacks embed an exploit during training that is activated by a test-time trigger, with model-agnostic and model-specific variants.Additional categories address transfer-learning and federated-learning settings.
- Defenses: Dataset-tampering defenses either detect poisoned instances or models, or train and correct models to resist or remove poisoning effects.Training-based approaches include robust training routines and post-hoc correction of corrupted models.
2 Training-Only Attacks
Training-only attacks manipulate training data without requiring post-deployment access to test instances, making them relevant when datasets are collected from potentially compromised online sources.
- Scope: Training-only attacks manipulate training data without needing to modify test instances after the victim model is deployed.They are especially salient when training data is harvested from potentially compromised online sources.
- Taxonomy: Figure 1 organizes training-only data-poisoning attacks by the methodologies used to construct them.The taxonomy is presented as a visual depiction of the attack families.
2.1 Applications of Data Poisoning
Data poisoning affects diverse applications, from targeted or untargeted model behavior to spam filtering, recommendation, reinforcement learning, facial recognition, and federated learning, while privacy mechanisms can create security trade-offs.
- Attack goals: Poisoning attacks may target particular inputs or individuals, or indiscriminately affect model behavior across many inputs.Venomave targets a specific person’s utterance in automatic speech recognition, while untargeted attacks seek broad impact.
- Spam filtering: SpamBayes can be poisoned by labeling legitimate-email vocabulary as spam, increasing spam scores for legitimate emails at test time.The attack exploits a naive Bayes spam filter rather than a neural network.
- Recommendation systems: Recommendation-system poisoning can degrade overall accuracy, promote a target item, or use training data designed to appear legitimate.Matrix-factorization recommenders are reported as vulnerable to these attacks.
- Privacy and security: Differentially private training limits the effect of modifying a small number of samples, but privacy-preserving aggregation can also let attackers mask malicious updates.The same privacy mechanisms can therefore provide resistance to poisoning while creating opportunities to hide manipulation.
- Other applications: Poisoning vulnerabilities extend to contextual bandits, stochastic bandits, online learning, facial-recognition systems, and federated learning.Attacks can alter rewards, promote suboptimal actions, poison scraped face images, or prevent screening of masked federated updates.
- Privacy and security: Secure aggregation masks each user’s federated update before transmission, making incoming models impossible to screen for corruptions.The passage frames this as a tension between user privacy and security.
2.2 Feature Collision Attacks
Feature collision attacks perturb base-class training images so their feature representations approach a target, exploiting transfer learning to induce targeted misclassification while preserving clean-label appearance.
- Core mechanism: Feature collision attacks perturb base-class training images so their feature-space representations move toward a chosen test target.The objective is to cause the target example to be misclassified into the base class.
- Core mechanism: The attack fixes a pretrained feature extractor and optimizes poison examples derived from corresponding original base-class samples.The optimization operates against a surrogate feature representation before victim-model training.
- Clean-label construction: The loss balances feature proximity to the target against pixel-space proximity to the original sample, preserving the clean-label appearance of the poison.Additional BlackCard terms explicitly encourage the base label and separation from the target in pixel space.
- Polytope variants: Polytope variants surround the target with poison features and constrain the target to the mean of poison vectors to improve attack reliability.Both polytope-based methods use model ensembles to improve black-box transferability, although feature collisions remain brittle when victim details are unknown.
- Transfer learning: Feature collision methods fit transfer learning because attackers can craft poisons on surrogate models while anticipating similar victim feature representations.The setting commonly freezes the feature extractor and fine-tunes a classification head on a smaller poisoned dataset.
- Transfer learning: An attacker can break many transfer-learned systems by crafting poisons that cause feature collisions across a large battery of standard models.The survey also distinguishes attacks that poison pretraining data so effects persist through later clean fine-tuning.
2.3 Bilevel Optimization
Bilevel optimization models poisoning as a search for training data that causes a trained model to misbehave, while practical methods approximate or bypass its expensive inner optimization. Modern approaches improve effectiveness and scalability across transfer-learning and from-scratch settings, but often require substantial computation.
- Direct bilevel methods: Bilevel poisoning methods simulate training and optimize poison data through the pipeline to produce a model that misclassifies a target image.The formulation can also target untargeted degradation by maximizing average test loss.
- Optimization challenges: Classical-model attacks require exact solutions to the inner optimization problem, whereas neural-network attacks approximate it through repeated gradient steps.Back-gradient descent differentiates through several inner SGD steps, which is memory intensive and was initially applied to a single-layer network.
- Modern methods: MetaPoison scales unrolling to realistic architectures and training processes, crafts poisons jointly, and outperforms earlier methods in fine-tuning and from-scratch regimes.It also demonstrated poisoning against the Google Cloud AutoML API, but its large-scale ensembling requires numerous GPUs.
- Modern methods: Witches’ Brew aligns poison-data gradients with a target loss, enabling targeted from-scratch poisoning on ImageNet and attacks against Google Cloud AutoML.Benchmarking found it strongest on from-scratch CIFAR-10, while Bullseye Polytope performed better on higher-dimensional Tiny-ImageNet.
- Optimization challenges: Generative models can produce additional poisons with inexpensive forward passes, but training the generator is costly and the resulting attacks may be less effective.These methods aim to mimic poisons generated by bilevel optimization.
- Alternative objectives: Gradient-based poisoning variants can instead induce vanishing training gradients or perturbations that make models ineffective or reduce overall test accuracy.TensorClog preserves high SSIM while disrupting training, and related generative methods pursue similar degradation objectives.
2.4 Label Flipping
Label flipping attacks poison models by changing training labels without altering data instances. Random or adversarially selected flips can affect robust learners and black-box linear models.
- Attack mechanism: Label flipping changes training labels while leaving data instances untouched, avoiding strange-looking artifacts that could alert the victim.These attacks are not clean-label attacks.
- Effectiveness: Adversarially chosen label flips can poison support vector machines, including learners trained with robust procedures.Earlier work considered both random and adversarial label flips.
- Effectiveness: Projected gradient ascent makes label flipping effective against black-box linear models including SVM, logistic regression, and LS-SV...
2.5 Influence Functions
Influence functions estimate how training examples affect learned parameters and predictions, providing an approximation to bilevel poisoning optimization. They are useful for classical models but do not fully capture dependencies in modern deep networks.
- Role in poisoning: Influence functions estimate the effect of infinitesimal training-data changes on resulting model parameters and can construct poisoning instances.They approximate solutions to the bilevel poisoning formulation.
- Applications: Influence functions can measure how removing a training point changes test loss and help create adversarial examples or correct mislabeled data.The method uses the Hessian of the loss at the trained parameters.
- Scope boundary: Influence-function methods have accelerated poisoning attacks and been applied to recommender systems, but they do not effectively capture data dependence in modern deep neural networks.The limitation is associated with highly non-convex loss surfaces.
2.6 Online Poisoning
Online poisoning extends data corruption to adversarial modifications made during dataset formation, including clean-label and block-wise attacks. Theoretical work characterizes targeted and untargeted vulnerability as a function of the perturbed-data fraction.
- Origins: Online poisoning originated in models where each data bit is perturbed independently with probability p.
- Threat models: Clean-label online attacks are called p-tampering attacks, and block-wise p-tampering allows entire training samples to be modified.
- Theory: Theoretical results establish targeted clean-label attacks for deterministic learners and untargeted attacks that degrade confidence in PAC learning.Quantitative vulnerability bounds depend on the portion p of the data that is perturbed.
2.7 Data Poisoning in Federated Learning
Federated learning introduces poisoning opportunities at multiple pipeline stages, including direct manipulation of local updates and coordinated attacks by colluding clients.
- Federated-learning adversaries can insert poisons at various training stages, and attacks need not be clean-label because victims cannot inspect attacker data.
- Late-injected label-flipping poisons are significantly more effective than early-injected poisons in federated learning.
- Model poisoning directly manipulates local models or gradient updates instead of modifying data or labels.
- Sybils are colluding agents whose attack effectiveness can depend on their number; FoolsGold reduces the influence of highly similar client contributions.
- Theoretical p-Tampering work for multi-party learning assumes the adversary knows updates generated by other benign parties.
2.8 Open Problems
Open problems include making poisoning attacks scalable, realistic, stealthy, comparable, robust across training choices, and capable of broader impacts.
- Bilevel optimization is more effective than feature collision for poisoning from-scratch neural-network training but is computationally expensive at industrial scale.
- Realistic training-only attacks should work with limited knowledge of the victim’s dataset or even the exact task.
- Truly clean-label attacks remain an open problem because existing methods often allow visibly corrupted poison images through large perturbation budgets.
- Fair comparison remains difficult because studies use varied experimental settings, and many attacks and threat models lack standardized benchmarking.
- Many poisoning methods are less effective under different architectures, optimizers, or augmentation strategies, motivating controlled robustness evaluation across hyperparameters.
- Broader poisoning objectives, such as affecting an entire subclass or demographic, remain largely unexplored beyond targeting individual images or all data.
3 Backdoor Attacks
Backdoor attacks poison models so a test-time trigger elicits adversary-chosen behavior while normal inputs retain ordinary behavior, spanning models, modalities, and threat models.
- Backdoor attacks exploit limited test-time input access to change model behavior across a broader range of inputs than training-only attacks.
- A trigger is an easily applied pattern, such as an image patch or phrase, that elicits a chosen behavior while the model behaves normally without it.
- End-to-end backdoor attacks commonly inject multiple triggered poison samples into training data, covering tasks including text, graphs, malware detection, and biological applications.
- Backdoors affect classifiers and generative models, including language models that produce specified or offensive text after a trigger phrase.
- Generative-model attacks must satisfy application-specific constraints, such as natural syntax in language or non-executable dead code in source-code models.
- Model-agnostic attacks use simple triggers without model knowledge and can therefore operate in black-box settings.
- Basic attacks mislabel natural triggered samples, while later work demonstrates effectiveness with few or imperceptible poisons, physical triggers, and unobtrusive text modifications.
4 Defenses Against Poisoning Attacks
Defenses against poisoning attacks detect compromised data or models, remove backdoors, or use robust training, but their applicability and security remain bounded by access and domain constraints.
- Defense methods detect poisoned data or models, repair trained models, or prevent poisoning through robust training procedures.
- Detection analyzes raw inputs, latent representations, or model behavior around selected inputs to distinguish poisoned from clean systems.
- Input-space outlier detection is intuitive but mainly effective for simple, low-dimensional domains; complex image and text data motivate latent-space methods.
- Outlier-based defenses can be bypassed by adaptive attacks that cluster poisoned inputs, and defense research remains concentrated in image classification.
- Latent covariance analysis can improve detection of standard backdoors over simpler feature-space ℓ2 distance metrics.
- STRIP mixes inputs with benign examples to detect trigger reliance in deployed models by monitoring prediction stability.
- Training-data-dependent detectors cannot operate when model training is fully outsourced, motivating model-only defenses such as trigger recovery and meta-classification.
- MNTD uses carefully crafted inputs and a meta-classifier to predict whether a model is backdoored, with apparent detection beyond its training architectures.
5 Conclusion
The article surveys data-poisoning and backdoor research, highlighting expanding vulnerabilities alongside machine-learning scale and the need for controlled comparisons.
- The authors systematically dissect research directions in data poisoning and defenses to provide a broad perspective on dataset security.
- The enumerated open problems reflect the authors’ interests and experience rather than an exhaustive account of outstanding challenges.
- Controlled comparisons are currently lacking in the data-poisoning and backdoor literature, motivating future datasets and benchmarks.
- The perspective aims to illuminate urgent industry security needs and support efforts to understand and close dataset vulnerabilities.