PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaPoison: Practical General-purpose Clean-label Data Poisoning

W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, Tom Goldstein

arXiv:2004.00225v2cs.LGcs.AIcs.CRcs.CVstat.ML

TL;DR

Deep-neural-network data poisoning is difficult because direct bilevel optimization is intractable and prior clean-label heuristics are limited in applicability. MetaPoison uses meta-learning to approximate the bilevel objective while crafting constrained poisons. It outperforms prior clean-label methods and succeeds across fine-tuning, training-from-scratch, alternative-scheme, and black-box settings.

  • Problem

    Direct bilevel optimization for deep-network poisoning is generally intractable, while prior clean-label methods have limited applicability and do not work when victims train from scratch.

  • Method

    MetaPoison uses a first-order meta-learning approximation with constrained poison perturbations, short-horizon surrogate unrolling, and ensembles across training epochs and initializations.

  • Results

    MetaPoison outperforms previous clean-label methods, succeeds on models trained from scratch, supports alternative poisoning schemes, and works on black-box Google Cloud AutoML models.

  • Takeaways & Limitations

    The method provides practical, general-purpose clean-label poisons that control victim behavior on specific targets across varied training settings.

  • Takeaways & Limitations

    The paper notes that keeping any single data source from contributing a large fraction of a training set or class helps keep a malicious contributor’s poison budget low.

Abstract

from arXiv · show

Data poisoning -- the process by which an attacker takes control of a model by making imperceptible changes to a subset of the training data -- is an emerging threat in the context of neural networks. Existing attacks for data poisoning neural networks have relied on hand-crafted heuristics, because solving the poisoning problem directly via bilevel optimization is generally thought of as intractable for deep models. We propose MetaPoison, a first-order method that approximates the bilevel problem via meta-learning and crafts poisons that fool neural networks. MetaPoison is effective: it outperforms previous clean-label poisoning methods by a large margin. MetaPoison is robust: poisoned data made for one model transfer to a variety of victim models with unknown training settings and architectures. MetaPoison is general-purpose, it works not only in fine-tuning scenarios, but also for end-to-end training from scratch, which till now hasn't been feasible for clean-label attacks with deep nets. MetaPoison can achieve arbitrary adversary goals -- like using poisons of one class to make a target image don the label of another arbitrarily chosen class. Finally, MetaPoison works in the real-world. We demonstrate for the first time successful data poisoning of models trained on the black-box Google Cloud AutoML API. Code and premade poisons are provided at https://github.com/wronnyhuang/metapoison

1 Introduction

Data poisoning modifies a small, imperceptible subset of correctly labeled training data to alter predictions on an unaltered target. MetaPoison addresses limitations of prior clean-label attacks by approximating bilevel optimization for deep networks and broadening attack settings and goals.

  • Threat and background: Clean-label poisoning imperceptibly perturbs correctly labeled training images while preserving their human-visible appearance.These perturbations can affect classifier behavior on a specific target image.
  • Limitations of prior methods: Bilevel poisoning optimization minimizes adversarial loss on a target through model weights obtained by minimizing training loss, but this is intractable for deep networks.Prior back-gradient methods that unroll training work for simpler models but not deep neural networks.
  • Attack objective: Figure 1 illustrates imperceptibly modifying a small fraction of training data so an unaltered parrot is classified as dog with high confidence while validation performance remains normal.The network is trained from scratch on the modified dataset.
  • Limitations of prior methods: Feature Collision is a dominant heuristic, but it requires knowledge of a stable feature extractor and therefore fails when victims train from scratch.It is also limited to objectives such as assigning a target the poison’s label.
  • MetaPoison contributions: MetaPoison approximates bilevel optimization and outperforms Feature Collision methods in fine-tuning settings.The authors also report successful clean-label poisoning from scratch, alternative poisoning schemes, and black-box Google Cloud AutoML poisoning.

2 Method

MetaPoison formulates poison crafting as constrained bilevel optimization and makes it tractable with short-horizon unrolling, ensemble surrogates, and re-initialization. The resulting updates seek poisons that steer victim training toward low adversarial loss across training stages and initializations.

  • 2.1 Poisoning as constrained bilevel optimization: The attacker optimizes poison images so an unaltered target receives a chosen adversarial label after training on mostly clean data.The poison set is added to the clean training set, with substantially fewer poison than clean images.
  • 2.1 Poisoning as constrained bilevel optimization: The outer objective minimizes adversarial loss after the network weights are obtained by minimizing standard cross-entropy training loss on clean and poison data.This nested dependence makes the formulation bilevel.
  • 2.1 Poisoning as constrained bilevel optimization: Poisons remain inconspicuous through ReColorAdv color remapping and additive perturbations bounded by perceptual constraints.The construction is xp = fg(x) + δ, with separate bounds on color remapping and additive noise.
  • 2.2 Strategy for crafting effective poisoning examples: MetaPoison approximates the intractable bilevel objective by unrolling only K SGD steps, then updating poisons using the resulting adversarial-loss gradient.The method uses K = 2 in the described work, and the two-step training trajectory is given explicitly.
  • 2.2 Strategy for crafting effective poisoning examples: The victim’s poisoned training trajectory is intended to reach weights with both low adversarial loss and low training loss.The poisons implicitly steer weights toward low adversarial loss while ordinary learning minimizes training loss.
  • 2.2 Strategy for crafting effective poisoning examples: Algorithm 1 alternates surrogate-model training, adversarial-loss evaluation, poison updates, averaging across models, and model re-initialization.The algorithm takes training data, a target, an adversarial class, perturbation thresholds, and multiple randomly initialized models as inputs.
  • 2.2 Strategy for crafting effective poisoning examples: An ensemble of partially trained surrogate models staggered by epoch makes poisons effective across insertion stages, while random re-initialization adapts them to diverse initializations.Each surrogate is vanilla-trained between poison updates, then reset after a sentinel number of epochs.
  • Implementation cost: MetaPoison requires 5760 forward-and-backward propagations per poison under the reported settings, while crafting 500 CIFAR-10 poisons takes about 6 GPU-hours.Loading pretrained surrogate checkpoints can reduce the reported crafting time to 5 GPU-hours.

3 Experiments

The experiments evaluate MetaPoison against prior fine-tuning attacks, from-scratch training, robustness across settings and architectures, and alternative poisoning objectives. Across these settings, the study measures attack success, transferability, and validation accuracy.

  • Comparison to previous work: MetaPoison reaches 100% attack success with about 25 poisons, only 0.05% of 50,000 training examples, in the fine-tuning comparison.This comparison uses the same fine-tuning conditions as Shafahi et al. and reports higher success at lower poison budgets.
  • Comparison to previous work: Unlike feature collision, MetaPoison poisons do not overlap the target in penultimate-layer feature space but push it across the class boundary toward the poison label.The visualization indicates a different mechanism from feature collision and suggests that neighborhood conformity defenses may be less effective.
  • Victim training from scratch: The study evaluates end-to-end poisoning from scratch using multiple target birds, victim models, architectures, poison budgets, and validation-accuracy measurements.Figure 4 includes 60 ConvNetBN victim models, attack success versus poison budget, and validation accuracy of poisoned models.
  • Robustness and transferability: Poisons remain overall robust under changes in learning rate, batch size, regularization, data augmentation, and victim architecture.Large changes in augmentation, learning rate, or batch size cause some but not substantial degradation, while transfer between ConvNetBN, VGG13, and ResNet20 is observed.
  • Versatility to alternative poisoning schemes: MetaPoison supports alternative objectives, including self-concealment and multiclass poisoning toward a chosen adversarial label.Self-concealment achieves 100% and 95% success for bird-bird and airplane-airplane pairs, while multiclass poisoning distributes a 10% total budget across all ten CIFAR-10 classes.

4 Conclusion

MetaPoison controls victim-model behavior on specific targets, outperforming prior clean-label methods and extending poisoning to models trained from scratch and industrial black-box services.

  • MetaPoison finds dataset perturbations that control victim model behavior on specific targets.
  • It outperforms previous clean-label poisoning methods on fine-tuned models and achieves considerable success on models trained from scratch.
  • MetaPoison enables novel attack schemes such as self-concealment and multiclass-poison.
  • The poisons work on industrial black-box machine-learning-as-a-service models.

5 Broader Implications

MetaPoison’s targeted attacks are difficult to mitigate because overall model behavior remains unchanged and the victim does not know which target input is compromised.

  • Targeted attacks are more difficult to mitigate than wholesale model-skewing attacks because overall model behavior remains unchanged.
  • The target input whose behavior is changed is not known to the victim.
  • Training-set governance should prevent any single data source from accounting for a large fraction of the dataset or a class.

A Poison crafting curves

The poison-crafting ablation compares surrogate-model loss with victim-model effectiveness across craft steps, showing that continued optimization can improve attacks after the surrogate loss plateaus.

  • Figure 8 compares crafting adversarial loss and victim-model attack success across the number of craft steps.
  • The crafting adversarial loss is averaged across 24 ensemble models, while victim evaluation fully trains 20 models from scratch at saved crafting stages.
  • After craft step 25, the surrogate adversarial loss decreases from approximately 9 to 4 but never falls below zero.
  • Victim-model effectiveness continues increasing through 200 craft steps even after the crafting loss plateaus.
  • The crafting curve alone cannot determine how well poisons perform during victim evaluation.

B Victim training curves

Victim-training analyses show that poisoning can drive target-specific attack loss toward success while preserving aggregate validation behavior, with outcomes varying across poison levels, class pairs, and target images.

  • Victim training curves: Training accuracy overfits CIFAR-10 after about 20k optimization steps, while validation accuracy looks similar for poisoned and unpoisoned datasets.
  • Victim training curves: With 5% poisoning, adversarial loss decreases to mostly negative values around −6, indicating near-perfect attack success.
  • Victim training curves: With 0.5% poisoning, adversarial loss hovers around zero, while unpoisoned models reach positive values around 12.
  • Differences across class pairs: Poisoning succeeds across all 100 CIFAR-10 poison-target class pairs, although vulnerability differs by target and poison class.
  • Differences in success rates amongst different targets: Success rates vary substantially among target images within the same class, making poisoning more dependent on the specific target image than the class pair.

E Ablation study on ensemble size

Ensembling surrogate models improves poisoning effectiveness, with success increasing as ensemble size grows.

  • Poisons crafted without ensembling are ineffective, while poisoning success rises as the surrogate ensemble size increases.

F Ablation study on reinitialization

Reinitializing surrogate networks improves poisoning efficacy compared with keeping their initializations fixed. MetaPoison also extends beyond single-target attacks to multi-target and indiscriminate objectives, with stronger results for more targeted variants.

  • 60% average success with reinitialization exceeded 51% with fixed surrogate initialization over 100 victim training runs.The authors describe this as a modest but significant enhancement in poisoning efficacy.
  • MetaPoison evaluates attack variants spanning multiple augmentations, distinct targets, class-specific errors, and fully indiscriminate errors.
  • Double-digit success rates were achievable for more targeted attacks, whereas indiscriminate attacks produced only a few percentage points of error increase.The authors attribute the weaker indiscriminate performance to adversarial-loss constraints involving many hold-out images.

H Ablation study on perturbation magnitude

Poisoning remains notable under minimal perturbation bounds, while larger additive and color bounds can increase success without significant perceptual change.

  • Even minimal perturbations of (ϵ, ϵc) = (2, 0.02) achieve notable poisoning in a 1% poison-budget dog-bird attack.The experiments use modest baseline bounds of (ϵ, ϵc) = (8, 0.04), with larger bounds offering higher success without significant perceptual change.

I Ablation study on number of unroll steps used during crafting

A small number of unroll steps is sufficient when combined with ensembling and surrogate-network reinitialization, and the crafting process is stable across random seeds. Subsampling from a larger poison set can outperform direct crafting at the target budget.

  • I Ablation study on number of unroll steps used during crafting: A single unroll step is insufficient, but 2–9 steps produce similar performance and 12 steps may improve success slightly.The authors recommend 2 unroll steps because they perform well while minimizing computational costs.
  • Poison crafting is generally stable across random seeds, with adversarial-loss histograms from different poison sets overlapping almost perfectly across 300 victim models.The stability experiment used six poison sets, each containing 500 poison dogs, and trained 300 victim models per set.
  • Subsampling 500 poisons from 5,000 produced more negatively skewed adversarial-loss histograms than crafting directly with 500 poisons.The authors report that subsampling to the desired budget achieves better performance and relaxes the requirement that the entire poison set be included.

L Experiments on ImageNet-2k (Dogfish) dataset

On Dogfish, MetaPoison succeeds with a single poison, while visualizations show how crafted poisons redirect target representations during training. Across layers, poisons adopt target-like features before the final classifier moves them toward the poison class.

  • L Experiments on ImageNet-2k (Dogfish) dataset: 100% success over the first 10 target fish images was achieved with one poison dog, while validation accuracy remained 82%.The experiment used last-layer transfer learning on a poisoned Dogfish dataset.
  • M.1 Feature evolution: A train-from-scratch attack using 50 poisons moves the target toward the poison distribution as training progresses.The penultimate-layer visualization shows poorly separated classes initially, followed by increasing class separation as earlier-layer extractors learn.
  • M.2 By layer: Across network layers, poisons remain near target-class regions until the final layer, where poisons and target move toward the poison-class cluster.The visualization covers the trained ConvNetBN at epoch 199 and uses the same projection method as Section 3.1.
  • M.2 By layer: The poisons develop target-like features from basic edges and color patches through higher-level semantic features, without explicitly colliding features.This provides a feature-space view of how poisoning redirects the target toward the poison side of the decision boundary.

N Further examples of data poisons

Additional CIFAR galleries show crafted dog and frog poisons paired with different target images. The examples use fixed poison parameters and display the first 24 poisons for three targets per class.

  • N Further examples of data poisons: The galleries use poison parameters ϵ = 8 and ϵc = 0.04, showing the first 24 poisons for the first three validation targets.Each gallery corresponds to a different target image, with poisons ordered according to the default CIFAR order.
  • N Further examples of data poisons: Including the top three rows of the dog-poison gallery causes the first bird target to be classified as a dog.The displayed dogs are CIFAR training images, while the bird targets are CIFAR validation images.
  • N Further examples of data poisons: Including the top three rows of the frog-poison gallery causes the first airplane target to be classified as a frog.The displayed frogs come from CIFAR training data, while the airplane targets come from the CIFAR validation set.
Loading 2004.00225v2…