PaperlayerView exact revision on arXiv

Source-linked AI summary

Subpopulation Data Poisoning Attacks

Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, Alina Oprea

arXiv:2006.14026v3cs.LGcs.CRstat.ML

TL;DR

Poisoning attacks threaten large, diverse training datasets, while existing attack settings can require extensive control or exact target knowledge. This paper proposes a modular subpopulation attack that selects a group and generates poisoned training points. The authors report attacks across modalities and models, improved targeted attacks, and limitations of existing defenses.

  • Problem

    Poisoning threatens large, diverse training datasets, while prior attacks can require control of a large training fraction or knowledge of exact targets.

    Source: §1 · pp. 1–2

  • Method

    The paper combines subpopulation selection with poison generation, using FeatureMatch or ClusterMatch and label flipping or suitable optimization methods.

    Source: §1.1 · p. 2, §9 · p. 22

  • Results

    The authors report effective attacks across tabular, image, and text data and end-to-end and transfer-learning models, alongside targeted-attack improvements and defenses that do not universally work.

    Source: §9 · p. 23

  • Takeaways & Limitations

    The authors conclude that some learners are inherently vulnerable under their assumptions and that existing poisoning defenses do not universally protect against subpopulation attacks.

    Source: §9 · p. 23

    Influence- and gradient-based optimization attacks are implemented only for image datasets because continuous feature modifications are difficult for categorical tabular and text data.

    Source: influence functions · p. 11

Abstract

from arXiv · show

Machine learning systems are deployed in critical settings, but they might fail in unexpected ways, impacting the accuracy of their predictions. Poisoning attacks against machine learning induce adversarial modification of data used by a machine learning algorithm to selectively change its output when it is deployed. In this work, we introduce a novel data poisoning attack called a \emph{subpopulation attack}, which is particularly relevant when datasets are large and diverse. We design a modular framework for subpopulation attacks, instantiate it with different building blocks, and show that the attacks are effective for a variety of datasets and machine learning models. We further optimize the attacks in continuous domains using influence functions and gradient optimization methods. Compared to existing backdoor poisoning attacks, subpopulation attacks have the advantage of inducing misclassification in naturally distributed data points at inference time, making the attacks extremely stealthy. We also show that our attack strategy can be used to improve upon existing targeted attacks. We prove that, under some assumptions, subpopulation attacks are impossible to defend against, and empirically demonstrate the limitations of existing defenses against our attacks, highlighting the difficulty of protecting machine learning against this threat.

1 Introduction

Poisoning is a concern for large, diverse training datasets, while prior attacks impose strong attacker capabilities or affect only narrowly specified targets. The paper introduces subpopulation attacks and reports effectiveness across datasets, as well as limits of existing defenses.

1 Introduction

  • Prior availability and backdoor attacks require control of a relatively large training-data fraction, while targeted attacks assume knowledge of exact targets and affect only a small set of points.The cited examples for availability and backdoor attacks use 10% or 20% of training data.Source: §1 · p. 2

1.1 Our Contributions

  • Subpopulation attacks seek to compromise a particular group while preserving performance elsewhere, without modifying test points or requiring knowledge of the exact model architecture and parameters.The authors connect the attack’s stealthiness to natural data points and relate its vulnerability to disparities affecting minority groups.Source: §1.1 · p. 2
  • With only 126 poisoned points, the attack induces 74% classification error on a CIFAR-10 subpopulation while maintaining accuracy similar to the clean model outside it; on UTKFace, 29 points increase error by 50% in one subpopulation.The attack affects both end-to-end-trained and transfer-learning models.Source: §1.1 · p. 2
  • The paper reports that subpopulation selection can improve targeted attacks; Witches’ Brew is 86% more effective when targeting points from a single subpopulation.The authors also state that 126 poisoned points induce 74% classification error on a CIFAR-10 subpopulation while maintaining similar accuracy outside it.Source: §1.1 · p. 2
  • The authors prove an impossibility result for models based on local decisions and report that TRIM/ILTM, SEVER, activation clustering, and spectral signatures do not universally defend against their attacks.The impossibility result identifies mixture models and k-nearest neighbors as inherently vulnerable.Source: §1.1 · p. 3

2 Background

The background introduces neural-network training and transfer learning, then distinguishes poisoning objectives and surveys related attack and defense families. It situates the paper’s approach as targeting subpopulations rather than only fixed points or inputs modified at test time.

2.1 Neural Networks Background

  • Neural networks combine linear layers and nonlinear activations; training commonly uses stochastic gradient descent, while transfer learning initializes a smaller-task model from one trained on a larger dataset.The paper distinguishes last-layer transfer learning from fine-tuning, where all network layers may be updated.Source: §2.1 · p. 3

2.2 Poisoning Attacks

  • Poisoning attacks add corrupted training data and are categorized as availability attacks that degrade overall accuracy, targeted attacks against specified points, or backdoor attacks triggered by modified test inputs.The related-work comparison organizes attacks by modality, training-point knowledge, test-time modification, and generalization to unknown points.Source: §2.2 · p. 4, Table 1 · p. 4

2.3 Related Work

  • Availability defenses such as SEVER, TRIM, and ILTM iteratively remove suspected outliers, using gradient bias or high loss to identify points.Backdoor defenses surveyed include latent-space outlier scoring, pruning with fine-tuning, and minimum-perturbation detection.Source: §2.3 · p. 5
  • Prior targeted attacks focus on specified points, whereas subpopulation attacks seek effects on a group; Witches’ Brew uses gradient matching and subpopulation identification can make targeted attacks more efficient.The paper also distinguishes natural-image backdoors that still require test-time modifications from its own attack setting.Source: §2.3 · pp. 5–6

3 Subpopulation Attacks

Subpopulation attacks target natural data points selected by a filter, aiming for high error within the selected group and low collateral damage elsewhere. The framework separates subpopulation selection from poison generation and assumes a constrained attacker with auxiliary data but without the victim’s training data or parameters.

3.1 Threat Model

  • The attacker adds a small number of training contaminants, cannot alter test inputs or access the victim’s training data or parameters, and uses a separate auxiliary dataset sampled from the same distribution.The threat model assumes knowledge of the learner’s loss and architecture, though the paper later removes the architecture assumption in Section 5.5.Source: §3.1 · p. 6

3.2 Definition

  • A subpopulation is a restriction of the input domain to points close under a distance function; the attack uses a filter to separate its target group from the remainder.For image similarity, the paper notes that distance in a trained model’s representation space can be more suitable than input-space ℓ2 distance.Source: §3.2 · p. 7
  • A successful attack seeks large target damage and small collateral damage on an unseen test set, thereby testing generalization to new group members rather than only prespecified points.The definition maximizes classification error within the selected group and can be modified for a specific target class.Source: §3.2 · p. 7
  • The attack interpolates between targeted poisoning, which selects a small fixed set, and availability poisoning, which selects the entire data domain.The paper frames its main setting as attacking a relatively small subpopulation while minimizing collateral effects elsewhere.Source: §3.2 · p. 7
  • The framework first identifies a vulnerable subpopulation with FeatureMatch or ClusterMatch, then generates poisons by label flipping or by influence- or gradient-based optimization.Label flipping samples points satisfying the filter and assigns them a label different from their original class; the filter is intended to limit collateral damage.Source: Figure 1 · p. 8

4 Methodology

The framework combines a filter for selecting a target subpopulation with a method for generating poisoned data. Its modular components allow attackers to select groups and generate poisons using different approaches.

4 Methodology

  • A complete attack pairs a subpopulation filter with an algorithm for generating poisoning data.The framework separates these components so they can be chosen or improved independently.Source: §4 · p. 8

4.1 Subpopulation Selection

  • Selecting a subpopulation that is separable from the rest of the distribution helps limit collateral misclassification.FeatureMatch and ClusterMatch are the two proposed selection approaches.Source: §4.1 · p. 8

4.1.1 FeatureMatch

  • FeatureMatch uses annotations finer-grained than or separate from labels to identify a chosen group, and tests attack performance on subpopulation test points.Such annotations can include race or age values for gender classification, or automobile color in CIFAR-10.Source: §4.1.1 · p. 9

4.1.2 ClusterMatch

  • ClusterMatch uses clustering on auxiliary data to identify a target group, potentially selecting a cluster aligned with the adversary’s goal and easier to attack.The authors use a neural-network representation and PCA for preprocessing, then KMeans; other clustering procedures may also work.Source: §4.1.2 · pp. 9–10

4.2.1 Label Flipping

  • Label flipping adds αm points sampled from the target subpopulation with a shared label different from their original label.The filter is intended to limit collateral when the target group is sufficiently separable, while label flipping seeks high target damage.Source: §4.2.1 · p. 10

4.2.2 Attack Optimization

  • Influence functions are extended from targeted poisoning, while Gradient Optimization (GO) uses a linear rather than quadratic approximation, trading effectiveness for efficiency.The GO objective targets increased loss on a target dataset; its optimization starts from label-flipped poisons and uses 50 gradient-descent steps, approximating the influence update with Hθ = I to avoid expensive Hessian operations.Source: §4.2.2 · pp. 10–11, influence functions · p. 11
  • Influence- and gradient-based optimization is implemented for image datasets only, because continuous edits are difficult for categorical, dependent tabular, and text features.For images, the attack requires valid pixels within [0, 255], not indistinguishability.Source: §4.2.2 · p. 10, influence functions · p. 11

4.3 Subpopulation Attack Framework

  • The attack framework proceeds from identifying subpopulations to selecting a target and generating poisons, with each component open to separate improvement.Poison generation may use label flipping or model-specific methods such as influence functions or gradient optimization.Source: §4.3 · pp. 11–12

5 Attack Experiments

Across four datasets and three modalities, the experiments test label-flipping attacks in end-to-end and transfer-learning settings, alongside fairness, transferability, and continuous-feature optimization. The results show substantial target damage across multiple settings, with attack effectiveness and collateral depending on the model and method.

5.1 Datasets and Models

  • The experiments span four datasets and three modalities; UTKFace gender classification excludes children under 15, leaving 20,054 images.The experiments compare small and large models on three of the datasets.Source: §5 · p. 12, §5.1 · p. 12

5.2 Label Flipping on End-to-end Training

  • On UCI Adult, ClusterMatch at α = 2 reaches 66.7% target damage in one subpopulation and averages 36.7% across five, and tends to outperform FeatureMatch when both apply.FeatureMatch reaches 25% target damage for one UCI Adult subpopulation at α = 2 and averages 19.5% over five subpopulations.Source: §5.2 · p. 14
  • On CIFAR-10, last-layer ClusterMatch reaches 32.7% target damage across the top five clusters, versus 18.7% using input features.Across 50, 100, 200, and 400 clusters, results show little significant difference; too few clusters can affect overall accuracy, while too many can leave very small target groups.Source: Table 3 · p. 13, §5.2 · p. 14

5.3 Label Flipping on Transfer Learning

  • Large-model ClusterMatch reaches 40.5% average target damage across 10 UTKFace subpopulations and 51.1% across 10 CIFAR-10 subpopulations.Across five subpopulations, UTKFace reaches 38.5% at α = 1 and CIFAR-10 reaches 61.6% at α = 1; collateral is typically around 1.5% for other datasets and models.Source: Table 6 · p. 16, §5.3 · p. 16

5.4 Fairness Case Study

  • FeatureMatch reduces accuracy from 91.4% to 76.7% for Black women high-school graduates on UCI Adult, illustrating the attack’s fairness implications.On UTKFace, the easiest-to-attack group’s accuracy falls from 100% to 60%.Source: §5.4 · p. 16

5.5 Subpopulation Transferability

  • ResNet-50 embeddings still yield 15–16% target damage across most CIFAR-10 subpopulations, with no more than 2% collateral damage.Architecture knowledge helps generate ClusterMatch subpopulations but is not necessary.Source: §5.5 · p. 17

5.6 Attack Point Optimization

  • For UTKFace with VGG-LL at α = 1, target damage averages 8.4% with label flipping, 15.5% with Gradient Optimization, and 38.6% with influence.Gradient Optimization does not improve average performance on VGG-FT and often decreases it; it is much faster than influence, which can time out on VGG-FT.Source: §5.6 · p. 17, Table 8 · p. 18

6 Improving Targeted Attacks Using Subpopulations

The section shows that selecting targets from ClusterMatch subpopulations can make fixed-target poisoning attacks easier than selecting targets arbitrarily. This holds for both influence-function and Witches’ Brew attacks.

6 Improving Targeted Attacks Using Subpopulations

  • ClusterMatch-generated subpopulations make fixed sets of k inputs easier to target than arbitrarily selected inputs, while the targeted attack algorithms remain unchanged.The section tests this complementarity using influence functions and Witches’ Brew.Source: §6 · p. 18

6.1 Influence Functions [KL17]

  • Top-5 attack success averaged 0.6 for ClusterMatch targets versus 0.24 for random selection; successful attacks took 250 versus 380 iterations to compromise targets.The original paper reported 2000 iterations and roughly four hours for successful attacks; ClusterMatch took roughly 30 minutes.Source: §6.1 · pp. 18–19, Table 9 · p. 19

6.2 Witches’ Brew [GFH+21]

  • 2.35x greater effectiveness on the 10 most effective attacks, and 86% better average performance across 24 trials, came from ClusterMatch target selection for Witches’ Brew.Targets were selected within per-class subpopulations and compared with randomly selected points from a single class.Source: §6.2 · p. 19, Table 10 · p. 19

7 Impossibility of Defenses

Under a simplified mixture-model learning framework, the paper proves that sufficiently small subpopulation poisoning attacks can defeat locally dependent learners. The result is conditional, and the authors note ways it may not reflect practice.

7 Impossibility of Defenses

  • For a dataset of size n from a k-subpopulation mixture, an attack of size at most n/k makes locally dependent learners return the poisoned-data result with probability below 1/2.If the smallest subpopulation has weight α, an attack smaller than 2αn suffices with probability at least 1 − exp(−αn/2).Source: §7 · p. 20
  • The result applies to learners making subpopulation-wide decisions; the cited framework includes k-nearest neighbors and mixture models, and suggests neural networks may share this structure.The authors say the result is more consequential for large k, representing more diverse data and smaller attacks.Source: §7 · p. 20
  • The authors caution that the result may not reflect practice when a learner avoids subpopulation-wide decisions, and it covers only purely algorithmic defenses.A carefully annotated, diverse validation set could help identify attacks, but requires knowing which subpopulations may be targeted.Source: §7 · p. 20

8 Empirical Analysis of Existing Defenses

The evaluation finds that existing availability and backdoor defenses do not reliably reduce subpopulation attack damage. It also illustrates how an availability defense can amplify an attack by removing real data from the targeted subpopulation.

8 Empirical Analysis of Existing Defenses

  • TRIM/ILTM and SEVER iteratively remove points scored as outliers using loss or gradient statistics, while the evaluation allowed five iterations and set the attack count to the poison count.The authors chose these settings to illustrate the defenses’ best possible performance under the evaluation.Source: §8 · p. 21
  • In the synthetic example, TRIM identified real points from the poisoned subpopulation as outliers, worsening the attack’s bias toward the poisoned class.Similar results were reported for SEVER.Source: §8 · p. 22, Figure 2 · p. 22
  • Across the evaluated datasets and models, availability defenses were inconsistent, while activation clustering and spectral signatures did not reliably reduce target damage.TRIM reduced damage for some UTKFace subpopulations but left it unchanged or increased it for others; activation clustering removed 25.7% of the training set while target damage remained fairly unchanged.Source: Table 12 · p. 21, §8 · p. 22

9 Conclusion

The paper concludes that subpopulation poisoning is a flexible threat that works across diverse data and model settings, and that existing defenses do not reliably prevent it. The authors connect this threat to fairness and robustness and identify defense development as future work.

9 Conclusion

  • FeatureMatch uses manual annotations and ClusterMatch automatically generates subpopulations from data; label flipping is generic, while optimization-based attacks can improve performance when suitable for the data modality.The attack does not require knowledge of the learner’s training data or architecture, nor modification of target test data.Source: §9 · p. 22
  • The attacks succeed on tabular, image, and text data, with models trained both end-to-end and through transfer learning.The paper also reports that subpopulation identification can improve existing targeted attacks.Source: §9 · p. 23
  • The paper’s impossibility result suggests some learners cannot defend against subpopulation attacks, and its evaluations find that existing poisoning defenses do not work universally.The authors identify protecting machine learning against this threat as a future challenge.Source: §9 · p. 23

A.1 Proof for Section 7

Under a noisy mixture model, the proof shows that subpopulation poisoning can defeat locally dependent learners that minimize binary 0-1 loss. It gives a bound based on the number or weight of subpopulations and explains the attack by flipping labels in the smallest one.

A.1 Proof for Section 7

  • The model uses k noisy subpopulations with distinct known supports, mixture weights, and subpopulation-specific label distributions; learners return a label for each subpopulation.Local dependence means each subpopulation's prediction uses only labels from that subpopulation.Source: §A.1 · pp. 29–30
  • An attack of size at most n/k suffices against locally dependent k-subpopulation mixture learners minimizing binary 0-1 loss.The theorem considers datasets drawn from a noisy k-subpopulation mixture distribution.Source: §A.1 · p. 30
  • With smallest subpopulation weight α, an attack of size less than 2αn suffices with probability at least 1 − exp(−αn/2).The bound is especially useful for mixtures with non-uniform weights.Source: §A.1 · pp. 30–31
  • The attack flips labels in the smallest subpopulation, making its majority label ambiguous to a learner that bases decisions only on that subpopulation.The proof uses this ambiguity to derive a contradiction between matching the clean classifier and flipping the subpopulation's decision.Source: §A.1 · p. 30
  • For n = 1000, α = 0.01, and k = 5, the Chernoff bound gives a poisoning size of 20 with probability greater than 99.3%, versus 200 by the pigeonhole bound.The comparison illustrates the tighter bound for non-uniform mixture weights.Source: §A.1 · p. 31

B Example ClusterMatch Subpopulations

ClusterMatch generates six UTKFace subpopulations that are not perfectly human-interpretable but show consistent trends. Examples include clusters with apparent age, gender, and facial-hair or skin-tone patterns.

B Example ClusterMatch Subpopulations

Loading 2006.14026v3…