Source-linked AI summary
Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, Tom Goldstein
TL;DR
This paper studies targeted clean-label poisoning attacks on neural networks, where correctly labeled training examples manipulate a classifier’s behavior on a chosen test instance. It introduces optimization-based feature-collision attacks and shows reliable poisoning with one image in transfer learning, while broader end-to-end settings require multiple poisons and watermarking.
Problem
Targeted poisoning attacks need to manipulate a specific test instance without requiring control over labels or noticeably degrading performance on non-targeted examples.
Method
The paper crafts realistic poison images that collide with a target in feature space, then uses multiple poison instances and a watermarking strategy for end-to-end training.
Results
The attack achieves a 100% success rate across 1099 transfer-learning targets, with average overall test-accuracy degradation of 0.2%.
Takeaways & Limitations
Correctly labeled poison images can be difficult to detect while controlling targeted classifier behavior, especially in transfer learning.
Takeaways & Limitations
A 100% success rate is not generally possible for transfer-learning tasks; the reported result depends on the InceptionV3 dog-vs-fish setting.
Abstract
from arXiv · showhide
Data poisoning is an attack on machine learning models wherein the attacker adds examples to the training set to manipulate the behavior of the model at test time. This paper explores poisoning attacks on neural nets. The proposed attacks use "clean-labels"; they don't require the attacker to have any control over the labeling of training data. They are also targeted; they control the behavior of the classifier on a $\textit{specific}$ test instance without degrading overall classifier performance. For example, an attacker could add a seemingly innocuous image (that is properly labeled) to a training set for a face recognition engine, and control the identity of a chosen person at test time. Because the attacker does not need to control the labeling function, poisons could be entered into the training set simply by leaving them on the web and waiting for them to be scraped by a data collection bot. We present an optimization-based method for crafting poisons, and show that just one single poison image can control classifier behavior when transfer learning is used. For full end-to-end training, we present a "watermarking" strategy that makes poisoning reliable using multiple ($\approx$50) poisoned training instances. We demonstrate our method by generating poisoned frog images from the CIFAR dataset and using them to manipulate image classifiers.
1 Introduction
The paper introduces targeted clean-label poisoning attacks on neural networks, which manipulate one chosen test instance without requiring control over training labels. It develops attacks that can evade detection and work in both transfer-learning and end-to-end settings.
- 1 Introduction: Targeted attacks manipulate a classifier’s behavior on one specific test instance rather than degrading performance broadly.Examples include changing one person’s identity in face recognition or controlling one chosen email’s classification.
- 1 Introduction: Classical poisoning attacks indiscriminately reduce test accuracy, whereas targeted attacks seek stealthier misclassification of selected examples.Prior DNN poisoning work included an 11% accuracy reduction when 3% of the training set was modified, even under strong defenses.
- 1 Introduction: Clean-label attacks use training examples that appear correctly labeled to human or certified reviewers, removing the need to control the labeling process.An attacker could place poisoned images online and wait for a data-collection bot to scrape them.
- 1 Introduction: The paper studies clean-label poisoning under transfer learning and, for the first time to the authors’ knowledge, under end-to-end retraining.The end-to-end setting is harder because deep networks can learn embeddings that separate the target from the poison.
- 1 Introduction: A 50-poison watermarking attack achieves success rates of up to 60% when all network layers are retrained.The approach is motivated by visualizations of why end-to-end poisoning is difficult.
2 A simple clean-label attack
The attack crafts a poison that looks like a base-class example in input space while colliding with a chosen target in the network’s feature space. An iterative optimization procedure balances these competing objectives.
- 2 A simple clean-label attack: The attacker selects a target and base instance, then minimally modifies the base instance into a poison intended to receive the base label while causing target misclassification.The poison is added to the training data before retraining the classifier.
- 2 A simple clean-label attack: The method uses penultimate-layer activations as feature-space representations encoding high-level semantic features.The function f propagates an input through the network to that layer.
- 2 A simple clean-label attack: The optimization makes the poison collide with the target in feature space while remaining close to the base instance in input space.This allows the poison to appear like a base-class example to a human labeler while embedding near the target class distribution.
- 2 A simple clean-label attack: The iterative procedure alternates a gradient-descent forward step minimizing feature-space distance to the target with a proximal backward step minimizing input-space distance to the base.The coefficient β is tuned to make the poison realistic enough to fool an unsuspecting human observer.
3 Poisoning attacks on transfer learning
Transfer learning makes targeted clean-label poisoning highly effective: one correctly labeled poison can induce target misclassification while leaving overall accuracy nearly unchanged. The attack works by substantially rotating the final-layer decision boundary.
- One-shot attack: A single correctly labeled poison instance causes target misclassification with 100% success in the dog-vs-fish transfer-learning task.The attack uses a pretrained InceptionV3 feature extractor and retrains only the final layer.
- One-shot attack: 100% attack success was achieved across 1,099 target images, compared with 57% for the influence-function method.The median misclassification confidence was 99.6%.
- One-shot attack: Overall test accuracy fell by only 0.2% on average, with a 0.4% worst-case drop from the original 99.5%.This preserves broad classifier performance while changing behavior on selected targets.
- Mechanism: 23 degrees was the average final angular deviation of the transfer-learning decision boundary after poisoning.Most parameter adjustment occurs during the first epoch, indicating that the poison substantially rotates the final-layer boundary.
- Robustness: The same 100% success rate is obtained when poison-base similarity uses an ℓ∞ bound of 2 instead of the main ℓ2 formulation.The bound is measured over an 8-bit dynamic range of 255.
- Scope: 100% poisoning success remains possible in a three-way dog-fish-cat task while clean-test accuracy remains 96.4%.The paper notes that the reported 100% transfer-learning success is not generally attainable and depends on the task setting.
4 Poisoning attacks on end-to-end training
End-to-end poisoning is harder because retraining can separate a single poison from its target, but watermarking and diverse multiple poisons make attacks effective. The mechanism pulls the target toward the base-class feature distribution without materially rotating the final decision boundary.
- 4.1 Single poison instance attack: A single poison fails under end-to-end training because retraining returns it to the base distribution while the target remains in its target distribution.This contrasts with transfer learning, where the final-layer boundary can rotate around the poison.
- 4.2 Watermarking: 50 watermarked poisons from diverse bases pull the target from its target-class feature distribution into the base-class distribution.The watermarks use a low-opacity target image blended into each base instance.
- 4.2 Watermarking: A 30% opacity target watermark is applied to base instances while keeping the resulting poison images visually distinct.The watermark preserves feature overlap between poison and target after retraining.
- 4.2 Watermarking: 50 poisons produce about 60% attack success for the bird-vs-dog task, with success increasing monotonically as poison count rises.Success requires the target to be classified specifically as the base class.
- 4.3 Outliers: 70% success is achieved against low-confidence airplane outliers using 50 poison frogs, 17% higher than for randomly chosen targets.The attacked outliers are correctly classified but have the lowest classification confidence.
- Mechanism: End-to-end poisoning combines optimization, diverse poisons, and watermarking; the paper’s ablation reports that all three are required for successful poisoning.The final-layer decision boundary remains nearly stationary, varying by fractions of a degree, while the target moves in feature space.
5 Conclusion
The paper develops targeted clean-label poisoning attacks that manipulate a classifier’s behavior on selected test instances while preserving performance on non-targeted examples. The attacks are difficult to detect and can be strengthened beyond transfer learning using multiple poisons and watermarking.
- Targeted clean-label attacks manipulate test-time behavior using correctly labeled, non-suspicious training data.They target a specific instance rather than broadly degrading classifier performance.
- Poison images collide with target images in feature space, making the two difficult for the network to distinguish.
- Transfer learning makes the proposed attacks extremely powerful, while multiple poison images and watermarking extend their effectiveness to more general settings.
- Poisoned training can increase robustness to base-class adversarial examples while causing unaltered target instances to be misclassified as the base class.
- The work highlights data reliability and provenance as important concerns because many neural networks use data sources that adversaries can manipulate.
A Illustrations of the poisoning scheme and of how the decision boundary will rotate after training with the poison instances
The poisoning scheme uses a few clean-label poison instances to alter the classifier’s decision boundary so a chosen target is misclassified. Unlike evasion attacks, the target itself is not modified; the attacker manipulates training data instead.
- A few poison instances can rotate the linear decision boundary in feature space to include the poison within the base-class region.
- The attack is illustrated as a clean-label poisoning process followed by a decision-boundary shift that changes classification of a target instance.
- Targeted clean-label poisoning differs from adversarial examples because it manipulates training data rather than slightly modifying the test target.
C Pixel bounded (i.e. L∞) poisoning attacks
The optimization formulation can enforce different similarity constraints between a poison and its base image. For L∞-bounded poisoning, clipping keeps every poison pixel within a specified distance of the base image.
- Different similarity measures can be enforced by changing the optimization problem’s similarity constraint.
- L∞ similarity is enforced by solving an optimization problem designed for pixel-bounded poison construction.
- Clipping after each gradient-descent update constrains every poison pixel to lie within ϵ∞ of the base image.The experiments use ϵ∞ = 2.
- The one-shot kill attack retains a 100% success rate under L∞-bounded poisoning attacks.
D One-shot kill attacks for multi-class transfer-learning scenario
The one-shot kill attack extends to a multi-class transfer-learning setting when the training problem is sufficiently underdetermined to permit overfitting on poison instances. In the reported three-class experiment, it achieved complete attack success while clean accuracy remained high.
- Transfer-learning attacks can generalize from binary to multi-class problems when training examples are fewer than model parameters, permitting poison-instance overfitting.The transfer-learning feature matrix is underdetermined in this setting.
- 96.4% test accuracy was achieved on clean examples in the three-class experiment.The experiment used 2,700 training examples and three classes.
- 100% attack success rate was achieved for 100 randomly sampled test targets using the one-shot kill attack.
- Outliers and points near the decision boundary are easier targets because changing the boundary there can require less classifier manipulation.
F Network architecture for CIFAR-10 classifier
The end-to-end CIFAR-10 experiments use a scaled-down AlexNet architecture, with poisoning outcomes shaped by target-dependent watermarks and base-instance choices.
- Network architecture: The end-to-end CIFAR-10 experiments attack a scaled-down AlexNet architecture.The clean-data network reaches 100% training accuracy and 74.5% test accuracy after 200 epochs.
- Poisoning setup: 60 poison instances successfully cause a bird target to be misclassified as a dog during end-to-end training.Each base instance receives a 30% opacity watermark of the target bird.
- Poison construction: Poison appearance depends on the target instance and the watermark opacity applied to base instances.The authors also describe placing target objects in innocuous image regions when multiple objects are present.
H Ablation study: How many frogs does it take to poison a network?
The ablation study shows that reliable end-to-end poisoning depends on combining multiple attack components and avoiding reuse of a single base image.
- Ablation study: The full poisoning procedure, combining all described methods, produces a successful attack.The corresponding feature representations before and after adversarial training are visualized in Figure 11.
- Base diversity: Reusing one base image for all poisons causes poisoning to fail because repeated adversarial instances make the network robust against adversarial examples.This condition is shown in row (b).
- Attack components: The ablation removes feature-collision optimization in one condition and watermarking in another.These are shown in rows (c) and (d), respectively.
- Attack outcomes: Bird-versus-dog attacks are more successful and visually less disturbed than airplane-versus-frog attacks.The bird-versus-dog experiment uses 60 poison examples, while poison appearance varies with target and watermark opacity.