PaperlayerView exact revision on arXiv

Source-linked AI summary

Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning

Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A. Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, Fabio Roli

arXiv:2205.01992v3cs.LGcs.AIcs.CR

TL;DR

Machine-learning deployment faces a poisoning threat when training data is not representative of test-time data. This paper develops a unified framework for threat modeling and poisoning attacks, aligns defenses with attacks, and identifies unresolved research challenges.

  • Problem

    Poisoning threatens machine-learning deployment when training data is not representative of data encountered at test time.

  • Method

    The paper proposes a unifying threat-modeling framework, a unified optimization framework for poisoning attacks, and an organization of corresponding defenses.

  • Results

    The work systematizes poisoning research, matches defenses with attacks, and shows the field’s historical development while deriving open questions.

  • Takeaways & Limitations

    The framework connects poisoning attacks, defenses, and threat models while highlighting open questions for further research.

  • Takeaways & Limitations

    The paper identifies unresolved challenges that continue to hinder thorough development of poisoning attacks and defenses.

Abstract

from arXiv · show

The success of machine learning is fueled by the increasing availability of computing power and large training datasets. The training data is used to learn new models or update existing ones, assuming that it is sufficiently representative of the data that will be encountered at test time. This assumption is challenged by the threat of poisoning, an attack that manipulates the training data to compromise the model's performance at test time. Although poisoning has been acknowledged as a relevant threat in industry applications, and a variety of different attacks and defenses have been proposed so far, a complete systematization and critical review of the field is still missing. In this survey, we provide a comprehensive systematization of poisoning attacks and defenses in machine learning, reviewing more than 100 papers published in the field in the last 15 years. We start by categorizing the current threat models and attacks, and then organize existing defenses accordingly. While we focus mostly on computer-vision applications, we argue that our systematization also encompasses state-of-the-art attacks and defenses for other data modalities. Finally, we discuss existing resources for research in poisoning, and shed light on the current limitations and open research questions in this research field.

1 INTRODUCTION

Poisoning attacks manipulate training data to compromise machine-learning performance, motivating a detailed survey of attack and defense strategies. The survey organizes the field through threat modeling, taxonomies, formalization, historical review, research resources, and open challenges.

  • Threat and attack categories: Poisoning attacks manipulate training data to degrade model performance on test data.Investigated categories include indiscriminate, targeted, and backdoor attacks.
  • Threat and attack categories: Targeted attacks seek misclassification of specific clean test samples, whereas backdoor attacks associate a trigger with an attacker-chosen class.Backdoor poisoning adds patterned, attacker-labeled samples that induce trigger-based misclassification at test time.
  • Survey framework: The survey presents a comprehensive framework for threat modeling poisoning attacks and categorizing defenses.It identifies practical scenarios enabling attacks and uses the framework to organize attacks and defenses.
  • Survey framework: Its systematic contributions include a unified optimization framework, attack-defense matching, historical analysis, and coverage of centralized learning.The survey also reviews learning settings, attack strategies, and training-time and test-time defenses.
  • Scope and resources: The paper reviews research resources and historical development while identifying limitations, open questions, scalable attacks, and evolving defenses.Although focused mostly on computer-vision classification, it discusses related work in audio, natural language processing, federated, unsupervised, and reinforcement learning.
  • Scope and resources: The survey categorizes around 45 computer-vision attack approaches and more than 70 defense approaches into six defense families.It formalizes poisoning optimization through bilevel programming and matches attacks with appropriate defenses.

2 MODELING POISONING ATTACKS AND DEFENSES

The survey models poisoning through learning settings, attacker objectives, knowledge, and capabilities, then uses this framework to organize attacks and defenses. It also identifies assumptions and challenges underlying existing strategies.

  • Threat-model dimensions: The survey systematizes poisoning attacks by learning setting, attacker goal, knowledge of the target system, and data-manipulation capability.Its framework also characterizes the defender’s goal, knowledge, and capability.
  • Learning settings: Training-from-scratch and fine-tuning rely on externally collected data, whereas model-training outsources the full procedure to an untrusted third party.Fine-tuning modifies only the classification layer, while model-training allows the untrusted party to train both feature mapping and classification functions.
  • Attacker’s goal: Attacker goals are classified by security violation, attack specificity, and error specificity, covering integrity, availability, privacy, targeted or indiscriminate effects, and class-specific or generic errors.The framework distinguishes which samples are affected from how their predictions are changed.
  • Attacker’s knowledge: Attacker knowledge spans training data, model details, test samples, and access mode, including white-box, transfer, and query-based black-box settings.The survey notes that knowledge of test samples can be crucial for some attacks, while sample-specific attacks may not generalize to other test samples.
  • Attacks and defenses: The framework supports categorizing defenses according to the attacks they prevent while exposing assumptions, current threat-model limitations, scalability challenges, and the continuing defense arms race.The survey presents this conceptual overview as a basis for understanding existing attack and defense strategies.

3 ATTACKS

The survey organizes poisoning attacks by the system property they compromise and reviews attack strategies within this framework. Figure 3 also separates defenses by intervention type and deployment timing.

  • The framework organizes existing attacks according to the attacker’s goal, knowledge of the target system, and capabilities.
  • Attacks are categorized by whether they compromise system integrity or availability.
  • Defenses are categorized by whether they sanitize data or modify the learning algorithm or model.
  • Training-time defenses are applied before deployment, whereas test-time defenses are applied after deployment; training-time interventions distinguish outsourced training from training-from-scratch and fine-tuning.
  • Figure 4 represents poisoning by showing how poisoning samples change the learned decision boundary.

3.1 Indiscriminate (Availability) Poisoning Attacks

Indiscriminate poisoning attacks aim to degrade model performance broadly or compromise system availability by manipulating training data. The survey covers label-flip, optimization-based, transferability, and scalable attack strategies, alongside their computational and setting-specific limitations.

  • Indiscriminate attacks inject or perturb malicious training samples to cause misclassification on clean validation samples.
  • In a street-sign example, poisoning samples rotate the classifier’s decision boundary and cause stop signs to be misclassified as speed limits.
  • Existing attacks primarily target training-from-scratch settings, while attacks against larger models can be computationally and memory demanding.
  • Label-flip attacks mislabel training samples without perturbing feature values, compromising the accuracy of models such as SVMs.
  • Bilevel Poisoning: Bilevel poisoning optimizes training perturbations to increase validation loss, but its inner risk-minimization and outer risk-maximization objectives make optimization challenging.
  • Transferability attacks craft poisoning samples with surrogate datasets or models and transfer them to another target model.
  • Gradient-based attacks can be computationally expensive, motivating heuristic, back-gradient, generative-model, and variable-reduction approaches for greater efficiency.

3.2 Targeted (Integrity) Poisoning Attacks

Targeted poisoning preserves overall system behavior while inducing errors on selected clean test samples. The survey covers bilevel, influence-based, feature-collision, and scalable clean-label attacks, emphasizing trade-offs among effectiveness, knowledge, and computational cost.

  • Targeted poisoning preserves availability and most clean-sample behavior while causing misclassification of specific target samples.
  • A conceptual example preserves accuracy for most clean samples but misclassifies a targeted stop sign as a speed-limit sign.
  • Targeted attacks optimize poisoning data against validation or test targets, with test-time realizations evaluated separately from optimization samples.
  • Bilevel Poisoning: Bilevel and influence-function approaches identify training points that affect selected predictions, but effective versions can remain too computationally demanding for DNNs.
  • Faster subpopulation attacks can reduce computational cost but do not achieve the same success rate across all subpopulations.
  • Feature Collision: Feature-collision attacks make poisoning samples approach target samples in feature space so fine-tuning associates the target with the poisoning label.
  • Feature Collision: Feature collision assumes a fixed feature extractor and becomes impractical when the attacker lacks exact knowledge of that extractor.
  • MetaPoison and related work extend clean-label targeted poisoning to training-from-scratch DNNs using more scalable approaches derived from bilevel formulations.

3.3 Backdoor (Integrity) Poisoning Attacks

Backdoor poisoning implants a trigger that causes targeted misclassification while preserving clean-sample behavior, creating an integrity violation. The survey organizes triggers by patch, functional, and semantic strategies and discusses stealth, deployment settings, and optimization.

  • Backdoor attacks induce misclassification for test samples containing a trigger without affecting classification of clean test samples.
  • A backdoored street-sign detector can classify ordinary signs accurately but predict speed limits for stop signs containing the trigger.
  • The survey distinguishes patch, functional, and semantic trigger families.
  • Trigger-based attacks select training data, blend a trigger into it, and assign labels corresponding to an attacker-chosen class.
  • Patch triggers use fixed visible patterns, whereas functional triggers distribute or adapt perturbations across the image to improve stealth.
  • Backdoor attacks may alter labels during poisoning, but clean-label variants use triggers such as reflections or invisible sinusoidal signals.
  • Semantic triggers use feasible real-world features such as facial expressions, filters, sunglasses, or earrings to induce misclassification.
  • Backdoor poisoning can be formulated as bilevel optimization that increases triggered validation error while minimizing error on clean validation samples.

3.4 Current Limitations

The survey identifies two major challenges: unrealistic threat models and the difficulty of scaling poisoning attacks and defenses to modern production systems. These limitations affect attack generalization, practical evaluation, and computational feasibility.

  • Threat-model realism: Existing threat models often poorly represent real production systems and attacker capabilities.The survey calls for assumptions that are less favorable to attackers and more realistic about controllable training data.
  • Threat-model realism: More realistic poisoning models should assume attackers control only a small fraction of training points.Controlling almost the entire dataset makes indiscriminate poisoning success less informative for production settings.
  • Threat-model realism: Evaluations that optimize and test targeted attacks on the same test samples can yield higher success rates without guaranteeing generalization.Such attacks may fail under minor input variations or uncontrolled physical acquisition conditions.
  • Threat-model realism: Physical-world conditions such as illumination, camera orientation, and trigger alteration remain underexplored in backdoor poisoning.Most studies use digital settings where implanted triggers remain nearly unaltered.
  • Computational complexity: Bilevel poisoning optimization remains computationally demanding because Hessian operations scale cubically in time and quadratically in space with model parameters.This limits attacks on modern deep models, although bilevel optimization can improve effectiveness and stealthiness.
  • Computational complexity: Bilevel-inspired attacks are more effective against backdoor defenses, but solving the bilevel problem efficiently remains an open challenge.The survey links faster optimization to more scalable evaluation and potentially more effective, stealthy attacks.

3.5 Transferability of Poisoning Attacks

Transferability allows poisoning attacks crafted with limited knowledge or surrogate models to affect unknown target classifiers. Matching surrogate and target complexity improves effectiveness, while limited representation knowledge can reduce it.

  • Black-box transferability: Transferability enables attacks crafted against surrogate learners or sampled surrogate data to mislead unknown target models.This is particularly relevant to black-box poisoning, where the attacker lacks exact knowledge of the target training setup.
  • Black-box transferability: Matching the complexity of surrogate and target models enhances poisoning attack effectiveness.Subsequent studies analyzed this property using surrogate learners.
  • Surrogate objectives: Smoother surrogate objectives can make optimization more effective when the target loss is nondifferentiable.A smoothed objective may reach effective or better local optima than the target function.
  • Representation transfer: Transferability can operate with limited feature-representation knowledge, but reduced knowledge comes at the cost of lower attack effectiveness.Feature-collision stability is also compromised when the representation changes.
  • Representation transfer: Attacking an ensemble of models encourages poisoning samples to transfer across multiple networks.This strategy addresses instability caused by changes in feature representations.

3.6 Unifying Framework

The survey unifies indiscriminate, targeted, and backdoor poisoning attacks within a generalized bilevel optimization framework. The framework represents attacker goals, data tampering constraints, and attack-specific test perturbations in one formulation.

  • Unified formulation: Indiscriminate, targeted, and backdoor attacks cause distinct violations but can be expressed through one bilevel optimization program.The framework uses attack-specific coefficients to encode the desired violation.
  • Attacker model: The generalized program optimizes poisoning perturbations under constraints reflecting attacker capabilities and knowledge of the victim model.Perturbations are applied to poisoning samples in D_p.
  • Attacker model: The perturbation set Δ can encode domain or feature constraints that improve attack stealthiness.An example is requiring an invisible backdoor trigger.
  • Attack-specific objectives: Test-data perturbation t is absent for indiscriminate and targeted poisoning but is predefined or optimized before training for backdoor poisoning.Unlike adversarial examples, the backdoor perturbation is not optimized at test time.
  • Open direction: Faster bilevel optimization could enable more effective and stealthy attacks while addressing the scalability limits of current strategies.The survey identifies efficient bilevel optimization as an open direction.

4 DEFENSES

The survey organizes poisoning defenses into categories spanning data sanitization, robust training, ensembles, model inspection, and test-input sanitization. Coverage is broad, but several attack classes and adaptive attacks remain insufficiently addressed.

  • Defense taxonomy: The defense review covers six classes, summarizes prior approaches, and matches defenses to attack strategies at training and test time.Tables 3 and 4 provide compact overviews and attack-defense correspondences.
  • Training-data sanitization: Training-data sanitization detects or removes suspicious samples using labels, features, clustering, outlier detection, or trigger reconstruction.These defenses generally require access to the training data, and sometimes clean validation data.
  • Training-data sanitization: Training-data sanitization has not yet mitigated indiscriminate clean-label bilevel attacks, semantical trigger backdoors, or bilevel backdoors.The survey reports these gaps in its attack-defense matching analysis.
  • Robust training: Robust-training defenses limit poisoning influence through robust optimization, regularization, augmentation, loss-based unlearning, preprocessing, pretraining, or differential privacy.These approaches alter the learning procedure and generally require access to model parameters.
  • Robust training: Robust training can only be implemented when the defender controls model training, such as training from scratch or fine-tuning.This creates a setting boundary for defenses that modify the learning algorithm.
  • Model and test-time defenses: Model inspection and test-data sanitization provide additional predeployment and test-time defenses, but adaptive backdoor attacks tend to break the same defenses.Inspection can use input patterns, latent representations, interpretability, or outlier detection to identify backdoored models.

5 POISONING ATTACKS AND DEFENSES IN OTHER DOMAINS

The survey focuses on poisoning in supervised learning, mostly computer vision, while noting related attacks and defenses across several other ML domains. It omits detailed treatment of those findings but presents its systematization as a starting point for understanding them.

  • The survey’s primary scope is poisoning in supervised learning, especially computer-vision applications.
  • Related poisoning attacks and defenses also appear in anomaly detection, computer security, audio, video, natural language, and graph-based ML.
  • The survey does not provide detailed descriptions of research findings outside its main scope because of space constraints.
  • Its systematization is intended as a useful starting point for understanding contributions in these other research areas.

6 RESOURCES: SOFTWARE LIBRARIES, IMPLEMENTATIONS, AND BENCHMARKS

Unified frameworks, libraries, datasets, and benchmarks support evaluation of poisoning attacks and defenses, but available implementations and benchmarking guidance remain limited. ART provides the broadest described implementation coverage, while attack success can depend strongly on experimental settings.

  • Unified test frameworks are important for evaluating and benchmarking poisoning attacks and defenses.
  • Only a few libraries provide implementations of poisoning attacks and defenses, despite many individual attack repositories.
  • ART offers the largest described collection, including indiscriminate, targeted, backdoor, and adaptive backdoor attacks plus several defenses.
  • SecML implements indiscriminate poisoning attacks against SVM, logistic, and ridge regression, while advBox provides indiscriminate and backdoor attacks on a toy problem.
  • The NIST competition database contains poisoned and unpoisoned models spanning image classification, object recognition, and reinforcement learning, and can generate new datasets.
  • Attack success depends highly on the experimental setting, and few works benchmark attacks and defenses or provide evaluation guidelines.

7 DEVELOPMENT, CHALLENGES, AND FUTURE RESEARCH DIRECTIONS

The survey traces the historical co-development of poisoning attacks and defenses, then identifies realism, scalability, robustness, and evaluation as continuing challenges. It highlights adaptive attacks, cross-domain connections, and guidelines for more realistic threat models as future directions.

  • Historical development: The survey reviews historical milestones for poisoning attacks and defenses through timelines, then derives challenges and future research directions.
  • Attack development: Indiscriminate poisoning evolved from heuristic attacks to bilevel optimization, while targeted and backdoor attacks later moved toward bilevel formulations.
  • Attack development: Recent attacks seek more realistic threat models, smaller or less visible data modifications, practical training settings, and improved scalability.
  • Defense development: Defense research spans sanitization, robust training, model inspection, model and test-data sanitization, and trigger reconstruction, with many approaches focused on backdoors.
  • Open challenges: Regularization and differential privacy have been reported as defenses, linking poisoning research to broader ML security and privacy questions.
  • Open challenges: Adaptive attacks are a historically relevant development omitted from both timeline figures and are discussed separately as a robustness challenge.
  • Open challenges: Realistic threat models remain an open question; proposed guidelines limit attacker knowledge and training-time control when evaluating poisoning safety.

8 CONCLUDING REMARKS

The conclusion frames data poisoning as a reliability concern for production ML systems. The survey addresses it through a broad systematization, unified formalization, resource review, historical analysis, and identification of open challenges.

  • The adoption of data-driven models in production demands rigorous reliability analysis and trustworthy deployment under poisoning threats.
  • The survey systematizes poisoning attacks and defenses, matches defenses to the attacks they prevent, and provides a unified bilevel-programming formalization.
  • It reviews benchmarking resources, traces the field’s historical development, and identifies open challenges and research directions.
Loading 2205.01992v3…