Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks

1 Introduction

• The attacker adds poison and camouflage points so the model initially behaves like one trained on clean data, then requests camouflage deletion to expose the targeted misclassification.The attack’s four stages are illustrated in Figure 1.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Camouflaged poisoning uses dynamic settings, including machine unlearning, to create a new attack vector against machine-learning models.The paper focuses on targeted attacks that aim to misclassify a specific test point.Source: §1 · pp. 1–2
• The camouflage set is constructed to undo the poison set’s impact, raising whether adding points alone can nullify a poisoning attack.The attack algorithm is based on gradient matching.Source: §1 · p. 2
• The authors demonstrate efficacy on SVMs and neural networks across CIFAR-10, Imagenette, and Imagewoof.Source: §1 · p. 2

1.1 Preliminaries

• Machine unlearning aims to make a model behave as if specified training points had never been included; retraining from scratch is the ideal deletion method but can take significant time.Source: §1 · p. 1, §1.1 · p. 3
• Poisoning attacks modify training data to harm test-time behavior, and this paper focuses on attacks targeting a specific test point.Common attacker constraints include adding few points and keeping introduced examples clean-label—unremarkable and correctly labeled to a human inspector.Source: §1.1 · pp. 3–4

• Marchant et al. (2022) poison unlearning systems to increase the time needed to unlearn a point, whereas this work targets misclassification of a test point.Source: §2 · p. 4
• Prior targeted-poisoning approaches include label flipping and watermarking, while clean-label attacks seek unobtrusive changes to features and labels.This work applies clean-label poisoning to both poison and camouflage sets.Source: §2 · p. 4
• Machine-unlearning research spans exact and probabilistic notions and multiple algorithmic settings; prior work also connects rapid deletion with drops in test loss.Source: §2 · p. 4
• The paper argues that its attack’s existence does not depend on the specific poison- or camouflage-generation procedure.The authors report preliminary success using the Bullseye Polytope technique and note that defenses against one procedure do not negate the attack’s contribution.Source: §2 · p. 5

3.1 Threat Model and Approach

• The attacker needs access to the victim’s model architecture, gradients, and a target sample, then adds poison and camouflage points and later requests camouflage removal.The target is previously unseen during training, and the attacker aims to preserve good performance on other validation samples.Source: §3.1 · p. 5
• After camouflage removal and retraining, the poison set can change the target’s prediction while the model’s earlier behavior and validation accuracy remain comparable to clean training.The camouflage set cancels the poisons during training; removing it exposes their effect.Source: §3.1 · p. 5

3.2 Gradient Matching for Efficient Poison Generation (Geiping et al., 2021)

• Gradient matching makes poison generation more tractable by matching poisoned-sample training gradients to the target’s adversarial-loss gradient at the clean-trained model.The method relaxes a condition that would otherwise need to hold for all model parameters.Source: §3.2 · p. 7

3.3 Camouflaging Poisoned Points

• The paper presents label flipping and gradient matching as two procedures for generating camouflage points that neutralize poison effects.Gradient matching is described as the main procedure for generating camouflages.Source: §3.3 · p. 8, §3.3.2 · p. 8

3.3.1 Camouflages via Label Flipping

• For binary classification with linear loss, label flipping pairs each poison point with the same image and opposite label, making the augmented training objective equivalent to clean-data training.The paper reports equal gradients and higher-order derivatives for the augmented and clean-data objectives.Source: §3.3.1 · p. 8

3.3.2 Gradient matching for generating camouflages

• Gradient-matched camouflages use samples with the target’s true label and perturbations optimized to match the target’s original-label loss gradient at the poison-trained model.The procedure relaxes the gradient condition to a fixed model trained on clean data plus poisons, and constrains perturbations by ε.Source: §3.3.2 · p. 9

4.1.1 Support Vector Machines

• On a linear SVM, attack efficacy exceeded 70% in most experiments, while clean-data validation accuracy was around 81.63% and changed little after adding poisons or camouflages.Table 2 covers Binary-CIFAR-10 and compares label-flipping and gradient-matching camouflages across threat models.Source: §4.1.1 · p. 10, Table 2 · p. 10

4.1.2 Neural Networks

• Camouflaging succeeded at least 70% of the time for VGG-11, VGG-16, ResNet-18, and ResNet-34, versus about 30% for MobileNetV2 and ResNet-50.Figure 3 reports efficacy across CIFAR-10 neural-network architectures and threat models.Source: §4.1.2 · p. 11, Figure 3 · p. 11

4.2 Evaluations on Imagenette and Imagewoof

• On Imagenette and Imagewoof, camouflaging succeeded in at least 50% of poisoning-successful cases, while modifying about 13% of training data coincided with validation-accuracy fluctuations up to 7%.The evaluation used VGG-16 and ResNet-18 across threat models.Source: Table 3 · p. 11, §4.2 · p. 12

4.3 Additional Experiments

• On CIFAR-10, additional experiments report robustness to random deletions of generated samples and training with data augmentation.The appendix also reports success across different threat-model values.Source: §4.3 · p. 12
• The attack transfers across models, meaning it succeeds when the victim model differs from the model used to generate poison and camouflage samples.The authors also report that poison and camouflage samples have similar feature-space distributions, which they say defeats sanitization defenses based on those distributions.Source: §4.3 · p. 12
• Most existing approximate-unlearning approaches were not evaluated because they were inefficient for the paper’s large-scale deep-learning settings, but the tested Amnesiac Unlearning method did not prevent the attack.The authors cite strong assumptions or high memory use as constraints on the unevaluated approaches.Source: §4.3 · p. 12

5 Discussion and Conclusion

• Camouflaged poisoning exposes a new attack vector in pipelines where training points can be unlearned, motivating attention to threats introduced by new ML functionality.The authors frame the attack as relevant to dynamic systems, including machine unlearning.Source: §5 · p. 12
• Preliminary Bullseye Polytope experiments also succeed, and the framework is described as accommodating other targeted-poisoning methods beyond gradient matching.The authors also say the attack extends straightforwardly to multiple targets by matching gradients from multiple target images.Source: §5 · pp. 12–13
• The attack is not black-box because poison and camouflage generation requires querying gradients of a trained model at new samples.The authors identify black-box attacks as an open research question.Source: §5 · p. 13
• The attack was not verified against other approximate unlearning methods because implementing them required unavailable resources.The experiments evaluate retrained-from-scratch models, which the authors describe as the objective approximate unlearning methods aim to emulate.Source: §5 · p. 13
• Further work could examine threats in online systems that add or remove points and camouflage indiscriminate or backdoor poisoning attacks.The authors also identify neutralizing attacks by adding points as an independent question.Source: §5 · p. 13

A Gradient Matching Algorithm for Efficient Poison Generation (Geiping et al., 2021)

• Algorithm 2 generates poisons by optimizing a perturbation against a gradient-matching loss, projecting updates onto a constraint set, and selecting the lowest-loss restart.Its inputs include a clean network, target, adversarial label, poison budget, perturbation bound, restarts, and optimization steps.Source: A Gradient Matching Algorithm for… · p. 19

B.1 Code

• The appendix provides ready-to-deploy Jupyter notebooks for Binary-CIFAR-10, CIFAR-10, Imagenette, and Imagewoof experiments, with public release planned for the final paper version.The notebooks cover a linear SVM for Binary-CIFAR-10 and various neural-network models for the other datasets.Source: §B.1 · p. 19

B.2 Experimental Setup

• The experimental setup tables map random seeds to target, poison, and camouflage classes and target IDs for CIFAR-10, Imagenette, and Imagewoof.The target IDs refer to each dataset's validation set.Source: §B.2 · p. 19, Table 4 · p. 20, Table 5 · p. 20

B.3.1 SVM Experiments

• Attack efficacy exceeded 70% in most Binary-CIFAR-10 experiments, while clean-data validation accuracy was around 81.63% and did not change significantly after adding or retraining with poison and camouflage samples.The reported attack efficacy applies across the experiments described in the passage.Source: §B.3.1 · p. 21
• The study evaluated both label flipping and gradient matching to generate camouflages, varying the threat model parameters ε, bp, and bc.The SVM used hinge loss and Scikit-learn's LinearSVC implementation.Source: §B.3.1 · p. 21
• Experiments used ten seeds of the form “kkkkkk” for k from 0 to 9, plus seed 99999; each clean-data SVM training run took 25–30 seconds.Poison and camouflage generation took about 40–50 seconds when bp = bc = 0.2%.Source: §B.3.1 · p. 21

B.3.2 Neural Network Experiments

• The neural networks used cross-entropy loss, mini-batch SGD, weight decay 5e-4, momentum 0.9, learning rate 0.01, batch size 100, and 40 training epochs.Each training run took about 45 minutes; poison and camouflage generation took about 1.5 hours.Source: §B.3.2 · p. 21
• For the architecture comparison, the first attack-success column counts successful poisoning trials and the second counts successful camouflage trials among those with successful poisoning.The threat-model experiments used five repetitions with eight restarts each and report mean success rate.Source: §B.3.2 · pp. 21–22
• Table 7 evaluates CIFAR-10 attack efficacy across neural-network architectures at ε = 16, bp = 0.6%, and bc = 0.6%, while Table 8 varies threat models on ResNet-18.The supplied passages describe the tables' evaluation scope but do not provide their cell values.Source: §B.3.2 · pp. 21–22, Table 7 · p. 22

B.4 Additional Details on Imagenette / Imagewoof Experiments

• Imagenette contains 13,394 images across ten classes, split into 9,469 training and 3,925 test images; Imagewoof contains 12,954 images, split into 9,025 training and 3,929 test images.Both datasets' images are resized and centrally cropped to 224 × 224 pixels for training.Source: §B.4 · pp. 22–23
• The Imagenette and Imagewoof attack evaluation uses VGG-16 and ResNet-18 across threat models listed in Table 3, with poison and camouflage generation by gradient matching.Models use cross-entropy loss and SGD for 50 epochs, with the learning rate decayed by γ = 0.9 each epoch.Source: §B.4 · p. 23

B.5 Visualizations

• The visualizations pair original and perturbed images: Imagenette and Imagewoof show them in alternating columns, while CIFAR-10 examples use top and bottom rows.Examples include ResNet-18 attacks on Imagenette, Imagewoof, and CIFAR-10, plus an SVM attack on Binary-CIFAR-10.Source: Figure 7 · p. 24, Figure 8 · p. 25, Figure 9 · p. 25

B.6.1 Ablation Study for Different Values of bp and bc

• The ablation varies camouflage fraction bc with poison fraction bp fixed at 1%, and vice versa, on ResNet-18/CIFAR-10.The reported comparisons are in Table 9; the supplied passages do not give its values.Source: §B.6.1 · p. 28, Table 9 · p. 28
• A separate comparison tests splitting a 2% budget into 1% poison and 1% camouflage against using 2% poison alone.The comparison is reported in Table 10 for CIFAR-10 on ResNet-18 with ε = 16.Source: §B.6.1 · p. 28, Table 10 · p. 28

B.6.2 Robustness of the Proposed Attack to Random Deletions

• Random-deletion experiments measure attack success after uniformly removing different amounts of generated poison and camouflage samples.The tests use CIFAR-10 on ResNet-18 with ε = 16 and report results in Table 11.Source: §B.6.2 · p. 28, Table 11 · p. 29

B.6.3 Transfer Experiments

• The generated poison and camouflage samples transfer across models, allowing an attack when the victim uses a different architecture from the one used to generate them.Figure 10 reports transfer experiments on CIFAR-10.Source: §B.6.3 · p. 29, Figure 10 · p. 29
• The “brewing” network is the architecture used to generate the samples, while the “victim” network is the architecture trained on the manipulated dataset.Source: §B.6.3 · p. 29
• Each reported transfer value is the fraction of three experiments per brewing/victim pair in which both poisoning and camouflaging succeeded.The experiments use seeds 2000000000–2000000011.Source: §B.6.3 · p. 29

B.6.4 Robustness to Data Augmentation

• 82% (no augmentation), 86% (Set 1), and 88% (Set 2) are the clean-data validation accuracies; augmentation makes poisoning harder and camouflaging easier.The augmentation conditions are no augmentation, 50% horizontal flipping, and flipping plus random rotations.Source: §B.6.4 · p. 30, Table 12 · p. 30

B.6.5 Similarity of Feature Space Distance

• The poison and camouflage feature-space distributions resemble clean images from their respective classes, making distance-based image sanitization ineffective against them.Figure 11 presents feature-space distances for the generated poison and camouflage set.Source: §B.6.5 · p. 30, Figure 11 · p. 31

B.6.6 Approximate Unlearning

• Most prior approximate-unlearning methods were infeasible to evaluate in this large-scale deep-learning setting because they require strong structural properties or large memory.The experiments so far assumed exact unlearning by retraining from scratch after each deletion request.Source: §B.6.6 · p. 30
• The attack continues to be effective against Amnesiac Unlearning, an approximate unlearning technique.Table 13 evaluates the attack using five randomly selected seeds.Source: Table 13 · p. 32

B.7 Multiple Targets Scenarios

• The attack continues to be effective with multiple targets, and a method that works better in that setting may increase its success.The experiments use gradient matching on CIFAR-10.Source: §B.7 · pp. 31–32, Table 14 · p. 32

C Alternative Approaches to Generate Poisons and Camouflages

• Bullseye Polytope maximizes similarity between poison and target representations, which implicitly minimizes their penultimate-layer gradient alignment in the described setting.In transfer learning, maximizing representation similarity can increase gradient alignment for the linear classifier.Source: C Alternative Approaches to Generate… · p. 32
• The BP experiments use a white-box setting with pre-trained ResNet-18 feature extractors, ε = 7, and 1,000 BP iterations for each poison and camouflage.The victim’s linear classifier is fine-tuned for 60 epochs on poisoned data and then on poisoned-plus-camouflaged data.Source: C Alternative Approaches to Generate… · pp. 32–33
• The camouflaged attack remains effective with Bullseye Polytope poisoning on CIFAR-10 across ten randomly selected seeds.Table 15 evaluates this clean-label targeted poisoning method.Source: Table 15 · p. 33