PaperlayerView exact revision on arXiv

Source-linked AI summary

Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

Jonas Geiping, Liam Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor, Michael Moeller, Tom Goldstein

arXiv:2009.02276v2cs.CVcs.LG

TL;DR

Targeted clean-label poisoning seeks to misclassify an unchanged test image by altering only a small fraction of training data, but earlier deep-network attacks were limited or costly. The paper matches poison and target gradient directions, showing reliable compromises of models trained from scratch, including ImageNet networks, while examining defense limitations.

  • Problem

    Earlier targeted poisoning attacks against deep networks were limited to simplified or transfer-learning settings, or were prohibitively expensive at large dataset scales.

  • Method

    The attack optimizes clean-label training-image perturbations to align averaged poison gradients with the adversarial gradient of a selected target.

  • Results

    The method reliably compromises randomly initialized deep models, including ImageNet models, while changing only 0.1% of data with perturbations below 8 pixel values in ℓ∞-norm.

  • Takeaways & Limitations

    Gradient matching enables targeted poisoning at unprecedented scale and effectiveness, making data poisoning a credible threat to large-scale deep learning systems.

  • Takeaways & Limitations

    Strong differential privacy can partially mitigate the attack, but reducing Poison Success below 15% requires sacrificing over 20% validation accuracy on CIFAR-10.

Abstract

from arXiv · show

Data Poisoning attacks modify training data to maliciously control a model trained on such data. In this work, we focus on targeted poisoning attacks which cause a reclassification of an unmodified test image and as such breach model integrity. We consider a particularly malicious poisoning attack that is both "from scratch" and "clean label", meaning we analyze an attack that successfully works against new, randomly initialized models, and is nearly imperceptible to humans, all while perturbing only a small fraction of the training data. Previous poisoning attacks against deep neural networks in this setting have been limited in scope and success, working only in simplified settings or being prohibitively expensive for large datasets. The central mechanism of the new attack is matching the gradient direction of malicious examples. We analyze why this works, supplement with practical considerations. and show its threat to real-world practitioners, finding that it is the first poisoning method to cause targeted misclassification in modern deep networks trained from scratch on a full-sized, poisoned ImageNet dataset. Finally we demonstrate the limitations of existing defensive strategies against such an attack, concluding that data poisoning is a credible threat, even for large-scale deep learning systems.

1 INTRODUCTION

Data poisoning exploits datasets assembled from potentially untrusted sources to alter trained models for targeted malicious outcomes. This work introduces an efficient clean-label attack that succeeds against deep networks trained from scratch, including full-sized ImageNet models.

  • Data poisoning makes imperceptible training-data changes to alter a model for a malicious objective, including targeted misclassification.
  • Targeted poisoning causes a predetermined test image to be misclassified, effectively creating a backdoor triggered by that specific image.
  • Previous targeted attacks often focused on linear classifiers or transfer learning rather than full end-to-end training of deep networks.
  • The proposed gradient-matching attack is orders-of-magnitude more efficient than a meta-learning formulation and succeeds more often.
  • 0.1% of data and perturbations below 8 pixel values in ℓ∞-norm suffice to compromise a ResNet-34 trained on ImageNet.

2 RELATED WORK

Targeted poisoning differs from evasion and backdoor attacks in its control over training data, test data, and target selection. Earlier poisoning methods either use detectable label or image manipulations, solve restricted bilevel problems, or target fine-tuning settings.

  • Targeted poisoning modifies training data while leaving test data unchanged and chooses target data in advance.
  • Unlike evasion attacks, poisoning acts before training; unlike backdoors, it does not require modifying target test images with a trigger.
  • Label flipping changes example labels, while watermarking superimposes target images onto training images; both can be detected through supervision.
  • Direct bilevel optimization is feasible for models such as SVMs and logistic regression but becomes intractable for deep neural networks.
  • Feature-collision methods are efficient but are designed primarily for fine-tuning scenarios rather than from-scratch training.

3 EFFICIENT POISON BREWING

The attack creates clean-label poisons by aligning their training gradients with the adversarial gradient of a chosen target. Differentiable augmentation, restarts, and efficient gradient computation improve transferability and scalability under a restricted threat model.

  • 3.1 THREAT MODEL: The attacker changes fewer than 1% of images within an ℓ∞ perturbation bound while preserving their semantic labels and lacking knowledge of initialization or training randomness.
  • 3.2 MOTIVATION: Poisoned examples mimic the gradient of a mislabeled target, so training on them also reduces the adversarial loss for that target.
  • 3.3 THE CENTRAL MECHANISM: GRADIENT ALIGNMENT: Because exact gradient matching across training is infeasible, the method minimizes negative cosine similarity between target and poison gradients.
  • 3.3 THE CENTRAL MECHANISM: GRADIENT ALIGNMENT: Each optimization step uses one differentiation through the parameter gradient instead of several unrolled optimization steps required by MetaPoison.
  • 3.4 MAKING ATTACKS THAT TRANSFER AND SUCCEED “IN THE WILD”: Differentiable augmentation, resampling, and restarts help poisoned perturbations transfer across training transformations and random initializations.
  • 3.4 MAKING ATTACKS THAT TRANSFER AND SUCCEED “IN THE WILD”: Model ensembles improve transferability but are expensive, so ImageNet experiments use cheaper restarts and data augmentation instead.

4 THEORETICAL ANALYSIS

The analysis explains why gradient-matching poisons can drive ordinary training toward low adversarial loss, while identifying the alignment assumption underlying the guarantee and supporting it empirically.

  • Gradient alignment: Gradient alignment measures cosine similarity between the adversarial-loss gradient and normal training-loss gradient.The analysis uses this quantity to connect poisoned training updates with adversarial descent.
  • Adversarial descent: Under bounded adversarial loss, Lipschitz gradients, and suitable positive step sizes, aligned training updates decrease adversarial loss.Proposition 1 formalizes adversarial descent for gradient-descent training.
  • Mechanism: The poisoning method makes the training-loss gradient a descent direction for adversarial loss, causing local convergence toward a stationary point of that adversarial objective.This provides the mechanism linking gradient matching to the original bilevel objective.
  • Assumption and limitation: The proposition assumes gradients remain almost always aligned, but the attack directly maximizes alignment only at a selected parameter vector rather than throughout every training trajectory.This is the strongest stated assumption and limitation of the theoretical argument.
  • Empirical evidence: Empirically, poisoned models maintain positive target-label gradient similarity throughout training, unlike clean models, whose normal training can increase adversarial loss.The reported comparison supports transfer of alignment beyond the parameter vector used to create the poison.

5 EXPERIMENTAL EVALUATION

Experiments show that gradient matching produces effective, scalable targeted poisons across CIFAR-10 and ImageNet, while exposing weaknesses in existing defenses. The attack remains effective across architectures and black-box deployment settings, but defense strategies impose substantial accuracy costs.

  • Evaluation protocol: The evaluation samples 10 random poison-target cases and tests each against 8 newly initialized victim models, reporting average poison success across these trials.The success criterion is correct classification of each target as its adversarial class.
  • CIFAR-10 evaluation: Differentiable data augmentation is crucial for scalable poisoning and can provide robustness comparable to a large model ensemble.On CIFAR-10, the proposed attack also succeeds with a single model rather than requiring an ensemble.
  • CIFAR-10 evaluation: Feature-collision objectives succeed only in the shallower network from scratch, while gradient matching outperforms MetaPoison on CIFAR-10 and is faster.The comparison uses a fixed brewing framework with one network and differentiable data augmentation.
  • Transferability: Using an ensemble of 2 ResNet-18, 2 MobileNet-V2, and 2 VGG11 models allows one poisoned dataset to compromise all evaluated architectures and generalize across them.Single-architecture poisons transfer to MobileNet-V2 but less effectively to VGG11 before ensemble construction.
  • ImageNet evaluation: 0.05% poison budget with ε-bound 8 compromises a randomly initialized ResNet-18 in 80% of trials, and the attack extends to MobileNet-v2 and ResNet50.For Cloud AutoML, a 0.1% ImageNet poison set with ε = 32 moved the target into the top-5 predictions in 5 out of 5 runs and top-1 in 1 out of 5.

6 CONCLUSION

The paper shows that gradient matching enables targeted poisoning against fully retrained deep networks at unprecedented scale, including ImageNet. It concludes that existing defenses are insufficient and that securing the entire data pipeline is necessary.

  • Gradient matching enables data poisoning attacks against fully retrained models at unprecedented scale and effectiveness.The attack is theoretically motivated, empirically evaluated, and succeeds against models trained on ImageNet.
  • The approach compromises models trained on ImageNet in realistic training scenarios.
  • The method is presented as a classification-focused test case, while related work considers applications including spam detection and face-unlock security.
  • Human supervision and outlier detection do not reliably prevent clean-label poisoning, while validation accuracy can remain unaffected.
  • The paper argues that data poisoning is best mitigated by fully securing the data-processing pipeline.

B EXPERIMENTAL SETUP

The experiments evaluate poisoning across randomized targets, poison classes, and reinitialized models, using specified datasets, architectures, augmentations, and hardware. ImageNet experiments include large-scale and cloud AutoML settings.

  • Experiments randomly select target classes, poison classes, target images, and poisoned images for repeated evaluation across reinitialized models.The usual design samples 10 settings, creates one poisoned dataset per setting, and evaluates it multiple times for CIFAR-10 and once for ImageNet.
  • The authors encourage sampling new target-poison pairs in larger numbers to reduce overfitting when computational resources permit.
  • Poison success is measured after complete retraining by whether the target image receives its adversarial class, not merely any incorrect label.
  • CIFAR-10 uses a 6-layer ConvNet alongside other architectures, with specified learning-rate schedules, stochastic gradient descent, and data augmentation.
  • ImageNet preprocessing uses resizing and cropping, horizontal flipping, and random crops during training.
  • The ImageNet AutoML experiment uploads slightly fewer than 950,000 training examples because of a 1-million-image platform limit.

C PROOF OF PROPOSITION 1

The proof analyzes gradient descent on adversarial loss under gradient alignment and smoothness assumptions. It derives conditions ensuring adversarial-loss decrease and convergence toward a stationary point.

  • The proof begins from the gradient-descent update and applies Lipschitz smoothness to bound the next adversarial-loss value.
  • Using the cosine identity, the analysis expresses adversarial-loss change through gradient norms and the angle between training and adversarial gradients.
  • Adversarial loss decreases for nonzero step sizes when the derived bound satisfies the proposition’s condition.
  • Summing the descent inequalities and using a lower bound on adversarial loss yields the limiting convergence argument.
  • If the gradient angle is below 90° except for finitely many iterates, with cosine bounded below by ϵ > 0, convergence to a stationary point follows.
  • Figure 5 estimates the bound using α_kL ≈ 1, while classical gradient descent converges only when α_kL < 1.

D POISONED DATASETS

The supplementary material provides poisoned datasets for replication, but only subsets of poisoned images rather than complete datasets.

  • The supplementary material provides poisoned datasets and code for replication, while omitting the full dataset and supplying only poisoned-image subsets.

E VISUALIZATIONS

The visualizations show that the attack remains difficult to distinguish from clean data, can target models under varied settings, and challenges existing defenses. Additional plots examine training dynamics, transfer, computational constraints, and differential-privacy countermeasures.

  • ImageNet and AutoML visualizations: Poisoned ImageNet samples are only barely distinguishable from clean data, including examples targeting randomly initialized ResNet-18 models.The same visual analysis includes a Google Cloud AutoML attack using ε = 32.
  • ImageNet and AutoML visualizations: A 0.1% poison budget with ε = 8 caused targeted misclassification of an otter image in a ResNet-18 trained on ImageNet.The poisoned images came from the Labrador Retriever class.
  • ImageNet and AutoML visualizations: A 0.1% poison budget with ε = 16 caused targeted misclassification for a randomly initialized ResNet-18 trained on ImageNet.The poisoned images again came from the Labrador Retriever class and targeted an otter image.
  • Training dynamics: The training plots compare adversarial and original target labels using loss and accuracy, while gradient plots track batch alignment and learning-rate drops.Measurements are averaged over epochs, and Figure 11 examines behavior before and after the first learning-rate drop.
  • Efficiency and defenses: Removing time and memory constraints exposes accuracy-versus-time comparisons with other poisoning approaches, but some successful CIFAR-10 attacks remain unsuitable for ImageNet-sized datasets.The comparison is shown for CIFAR-10 with a 1% budget and ε = 16 on ResNet18.
  • Efficiency and defenses: Outlier detection was only marginally more successful than random guessing, while attackers can counter differential privacy when its gradient noise is known.The counter-attack adds matching gradient noise and clipping, redrawing noise whenever the objective is evaluated.

F.5 ABLATION STUDIES - REDUCED BREWING/VICTIM TRAINING DATA

The ablations test reduced attacker knowledge, reduced victim exposure, network-width effects, optimization steps, and loss choices. Results show that the method can tolerate partial data overlap, while its strength depends on optimization and gradient-magnitude invariance.

  • Reduced brewing and victim data: The ablations reduce either the training set known to the attacker, the poisons included by the victim, or both.These settings model situations where victims scrape only some attacker-contributed images or attackers lack the full victim dataset.
  • Reduced brewing and victim data: The victim-training-data ablation directly measures how poisoning success changes when randomly selected clean and poisoned data are removed.A second regime also reduces the data used for pretraining before selecting the victim subset.
  • Method ablations: Cosine similarity is preferred to Euclidean loss because its invariance to gradient magnitude enables stronger poisoned datasets in the reported experiments.For widths smaller than 16, Euclidean loss dominates, but its effectiveness does not increase with width.
  • Method ablations: Using 50 optimization steps instead of 250 causes a significant loss in average poison success.The Carlini-Wagner surrogate does not help in this setting.

F.7 TRANSFER EXPERIMENTS

Transfer experiments test whether poisons crafted with one architecture remain effective against other architectures and whether poisoning multiple targets changes the budget-success trade-off. The results support cross-architecture transfer but show reduced per-target success as targets multiply.

  • Transfer experiments: Poisons crafted with one architecture can transfer and cause targeted misclassification in other networks.The ImageNet transfer experiments brew poisons with varied networks and test them against other networks.
  • Transfer experiments: Direct-transfer experiments use a 0.1% budget and ε-bound of 16 while training only on the brewing network, without victim-architecture knowledge.The figure describes transferability to unknown architectures.
  • Multiple targets: With a fixed 1% poison budget, the method can successfully poison more than one target, but each target receives an increasingly smaller share of the budget.The targets are randomly drawn and not semantically similar aside from their shared class.
  • Multiple targets: Near-100% average poison success for a single target is not reached when optimizing multiple targets, even after increasing the budget.Increasing the budget can nevertheless increase total misclassifications, including a score of 290 for 16 targets at 4%.
  • Clean validation accuracy: The poisoning attack does not significantly alter clean validation accuracy, with the CIFAR-10 baseline showing a drop on the order of 0.1%.The baseline uses ResNet-18 models, ε = 16, and a 1% budget.
Loading 2009.02276v2…