Source-linked AI summary
Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attacks
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, Tom Goldstein
TL;DR
Data poisoning methods are difficult to compare because reported success depends heavily on inconsistent and sometimes unrealistic evaluation settings. The paper builds standardized benchmarks across training and transfer-learning scenarios, finding that common practices can substantially reduce attack effectiveness while different attacks remain strongest in different settings. These benchmarks support fairer comparisons, though the study focuses on perceptual similarity rather than adaptive evasion of detection or defenses.
Problem
Poisoning attacks have not been evaluated consistently or realistically, and their reported success can depend on architecture, training protocol, and other setup choices.
Method
The paper benchmarks six image-classifier poisoning attacks across standardized from-scratch, white-box, and black-box transfer-learning settings using randomized trials and common training practices.
Results
Common practices such as SGD, data augmentation, and realistic victim architectures reduce attack effectiveness; for example, FC and CP success rates fall to 51.00% and 19.09% with augmentation.
Takeaways & Limitations
Public standardized benchmarks enable meaningful comparisons of poisoning methods and help identify where existing attacks may fail or pose practical concern.
Takeaways & Limitations
The evaluation emphasizes perceptual similarity under ℓ∞ constraints, while adaptive attacks designed to avoid automated detection or defenses remain relatively unexplored.
Abstract
from arXiv · showhide
Data poisoning and backdoor attacks manipulate training data in order to cause models to fail during inference. A recent survey of industry practitioners found that data poisoning is the number one concern among threats ranging from model stealing to adversarial attacks. However, it remains unclear exactly how dangerous poisoning methods are and which ones are more effective considering that these methods, even ones with identical objectives, have not been tested in consistent or realistic settings. We observe that data poisoning and backdoor attacks are highly sensitive to variations in the testing setup. Moreover, we find that existing methods may not generalize to realistic settings. While these existing works serve as valuable prototypes for data poisoning, we apply rigorous tests to determine the extent to which we should fear them. In order to promote fair comparison in future work, we develop standardized benchmarks for data poisoning and backdoor attacks.
1. Introduction
Data poisoning threatens machine-learning systems by manipulating training data, a concern amplified by large, web-scraped datasets that are difficult to vet. This paper develops standardized benchmarks to compare poisoning attacks under more consistent settings.
- Data poisoning lets an attacker control a machine-learning system’s behavior by manipulating its training data.
- Deep-learning systems are especially exposed because they require large datasets often scraped from insecure or unknown sources.Open Images and Amazon Products contain approximately 9 million and 233 million scraped samples, respectively.
- Industrial datasets are often not manually inspected and may accept contributions from malicious actors, raising concern that corrupted data can produce faulty models.
- A survey of 28 industry organizations found greater fear of data poisoning than of other adversarial-machine-learning threats.
- The benchmark focuses on image-classifier attacks that modify data and share a targeted-misclassification objective.
- The proposed benchmarks cover from-scratch training plus white-box and black-box transfer learning, while constraining poisoned images to small perturbations.The benchmarks are publicly available for evaluating existing and future attacks.
2. A Synopsis of Triggerless and Backdoor Data Poisoning
The paper distinguishes triggerless and backdoor poisoning by their inference-time behavior and evaluates representative attacks under a shared targeted-misclassification framework. It describes feature-collision, convex-polytope, clean-label-backdoor, and hidden-trigger methods.
- The benchmark studies targeted misclassification, using modified base-class images to make a target sample or class be classified as a specified other class.
- Backdoor attacks add a trigger at inference and misclassify any image containing it, whereas triggerless attacks target an individual image without an inference-time trigger.
- The study focuses on clean-label and hidden-trigger backdoors, whose poisons are optimized without noticeable patches.
- Triggerless attacks: Feature Collision crafts small perturbations so poison and target feature representations lie extremely close together.
- Triggerless attacks: Convex Polytope crafts poisons whose feature representations form a convex combination of the target’s feature representation.
- Backdoor attacks: Clean Label Backdoor first computes an adversarial perturbation for each base image, then adds a patch to generate the final poisons under an ℓ∞ constraint.
- Backdoor attacks: Hidden Trigger Backdoor keeps poisons close to base images while colliding in feature space with a patched target-class image.
3. Why Do We Need Benchmarks?
Existing poisoning evaluations vary across architectures, training protocols, class choices, and trial designs, making comparisons difficult to interpret. The paper addresses this by reproducing prior settings and systematically varying evaluation components with randomized trials.
- Prior studies use inconsistent architectures, target/base pairs, and experimental breadth, producing performance comparisons that are difficult to interpret.
- Some comparisons are unfair because attacks are evaluated with different constraints, such as CP’s ℓ∞ constraint versus FC’s ℓ2 penalty.
- Each trial samples one of ten checkpoints and randomly selects the target image, base class, and base images.Section 4 figures average results across 100 trials.
- The authors recreate original tests in a common framework, then change one evaluation component at a time to expose sensitivity to setup.
- Table 1 shows substantial variation in normalization, augmentation, optimizer, learning setup, threat level, and ensemble choices across original studies.
4. Just How Toxic Are Poisoning Methods Really?
The evaluation shows poisoning attacks are highly sensitive to experimental design, with effectiveness often declining under more realistic architectures, training procedures, transfer settings, and perturbation constraints. These results support standardized, thorough comparisons rather than rankings based on isolated baseline setups.
- Training procedure: SGD training and standard data augmentation greatly reduce attack effectiveness, with FC and CP success rates falling to 51.00% and 19.09%, respectively.The original FC and CP evaluations used ADAM-pretrained victims, whereas SGD with momentum and augmentation are common practical choices.
- Victim architecture: Victim architecture matters: attacks tested on less common AlexNet or narrow ResNet variants can perform worse against ResNet-18, with HTBD success as low as 18%.The ResNet experiments illustrate that attack performance depends strongly on architecture selection.
- Dataset size: Attack effectiveness changes substantially with dataset size, even when the poisoned fraction remains fixed at 1%.The authors report large dataset-size effects and intersecting FC and CP performance curves, so percentage-only budgets do not support fair comparisons.
- Black-box transfer: Black-box evaluation sharply reduces performance: FC, CP, and HTBD each achieve success rates below 20%.Poisons crafted with baseline methods were evaluated on victim models using architectures different from those used for crafting.
- Target robustness: Triggerless attacks are highly target-specific, with FC succeeding only 7% of the time after horizontally flipping the target image.This tests an upper bound on robustness when the exact target image is unknown.
- Backdoor and perturbation sensitivity: Backdoor success strongly correlates with patch size, while constraining poison perturbations to ℓ∞ radius 8/255 diminishes every attack’s effectiveness.The findings motivate standardized patch and perturbation constraints for fair comparison.
5. Evaluation Metrics for Dataset Manipulation
The benchmark standardizes poisoning evaluations across datasets, training regimes, architectures, and attacker knowledge settings. It uses reproducible image assignments, bounded perturbations, and shared evaluation procedures to enable fair comparisons.
- Benchmark design: The benchmark covers backdoor and triggerless poisoning attacks in standardized problem settings.Its deviations from original attack settings are chosen to preserve the original threats while improving comparability.
- Reproducibility constraints: Poison examples remain within an ℓ∞-ball of radius 8/255 around their corresponding base images, with seeded random target/base assignments.The same assignments are used for every method, reducing variation from image selection.
- Evaluation settings: Transfer learning freezes a clean pretrained feature extractor while training a poisoned linear classification head on disjoint data.The benchmark evaluates both white-box and black-box transfer learning scenarios.
- Evaluation settings: Training-from-scratch benchmarks train networks from random initialization on datasets containing poison examples.CIFAR-10 permits 500 poisons, while TinyImageNet permits 250.
- Datasets and budgets: CIFAR-10 uses 2,500 fine-tuning images with 25 poisons, while TinyImageNet fine-tuning permits 250 poison images.CIFAR-10 models are pretrained on CIFAR-100; TinyImageNet models are pretrained on its first 100 classes and fine-tuned on the second half.
- Reporting: Table 3 reports benchmark success rates as percentages, with the best performance in each column highlighted.Across six attacks, no single method is strongest in every setting.
6. Conclusion
The authors argue that poisoning risk requires fair comparison because methods vary widely in efficacy and some may not pose practical threats. Their benchmarks provide a common basis for comparing attacks and identifying where they fail.
- Practical significance: Some claimed poisoning methods do not pose a practical threat, while recent methods remain concerning to practitioners.The conclusion frames the field as requiring fair comparison rather than assuming uniform danger.
- Benchmark rationale: The diversity of attacks and difficulty ordering them by efficacy motivate a diverse set of benchmarks.The benchmarks are intended to compare attacks on a level playing field.
- Future use: The benchmarks are intended as a standardized test problem for future poisoning methodologies.The authors position them as a continuing evaluation resource as stronger attacks emerge.
A.1. Technical Setup
The technical setup specifies uncertainty estimation, fallback handling for sparse outcomes, and standardized hyperparameter choices for model training.
- Confidence intervals: Confidence intervals use a radius of one standard error around the observed success probability.The standard error is E = sqrt(p̂(1 − p̂)/N), where p̂ is observed success probability and N is the number of trials.
- Confidence intervals: When fewer than five successes or failures are observed, p̂ is set to 1/2 to upper-bound the standard error.This rule makes uncertainty estimates conservative for sparse outcomes.
- Training configuration: Models are trained using one of seven named hyperparameter sets.For SGD, the setup uses momentum 0.9, batches of 128 images, and weight decay 2 × 10^-4.
A.2. Baselines
The appendix defines implementation settings for four poisoning baselines and reports baseline performance over repeated trials with standardized uncertainty reporting.
- Evaluation protocol: Baseline performance averages results over 100 independent trials with confidence intervals of one standard error.Each trial randomly selects one of ten pretrained models for the specified architecture.
- Feature Collision: Feature Collision uses an unconstrained AlexNet-based attack with an ℓ2 optimization penalty and 1,200 maximum crafting iterations.Poisoned-data fine-tuning runs for 20 epochs with ADAM.
- Convex Polytope: Convex Polytope uses a normalized ResNet-18, constrains perturbations to ε = 25.5/255, and fine-tunes for 10 epochs.Poison crafting uses ADAM for up to 4,000 iterations or until the CP loss reaches 1 × 10^-6.
- Clean Label Backdoor: Clean Label Backdoor trains a narrow ResNet from scratch using adversarial perturbations generated with 20-step PGD.The perturbations use step size 4/255 and ε = 16/255.
- Hidden Trigger Backdoor: Hidden Trigger Backdoor uses a normalized modified AlexNet and patches the target image with an 8 × 8 patch.Poison crafting uses SGD for up to 5,000 iterations with ε = 16/255.
A.3. Training Without SGD or Data Augmentation
The authors add normalization and augmentation to pre-training and evaluate FC and CP under both ADAM and SGD, exposing sensitivity to training choices.
- FC and CP are evaluated with normalization and augmentation added to their pre-training processes.These modifications extend the original attack evaluations.
- Table 4 measures attack behavior with normalization and augmentation using ADAM.
- Table 5 measures attack behavior with normalization and augmentation using SGD.
A.4. Victim Architecture Matters
The benchmark tests attacks on ResNet-18 victim models, revealing whether results depend on the victim architecture used in the baseline.
- Each method is tested on ResNet-18 victim models.
- CP shows no change from its baseline because the baseline already uses a ResNet-18 victim.
- The architecture test isolates victim-model effects by holding the victim architecture to ResNet-18.
A.5. “Clean” Attacks Are Sometimes Dirty
Across standardized tests, poisoning effectiveness depends strongly on perturbation constraints, dataset scale, victim models, target variation, and attack design choices, limiting broad claims about attack strength.
- A.5. “Clean” Attacks Are Sometimes Dirty: Using ε = 8/255 constrains the ℓ∞ perturbations tested for every attack.HTBD shows no change because this value matches its baseline.
- A.5. “Clean” Attacks Are Sometimes Dirty: 1% poisoning does not produce attack strength that scales generally as dataset size and poison count increase together.The experiment ranges from 5 poisons and 500 training images to 500 poisons and 50,000 training images.
- A.5. “Clean” Attacks Are Sometimes Dirty: In the exact benchmark evaluation setting, attacks scale differently with dataset size, so percentage-only poison budgets do not support fair comparison.
- A.5. “Clean” Attacks Are Sometimes Dirty: Black-box transfer sharply lowers performance for every method except CLBD.CLBD is treated as black-box despite being intended for training from scratch because its crafting and testing architectures differ.
- A.5. “Clean” Attacks Are Sometimes Dirty: Flipping target images severely impairs triggerless attacks, supporting a lower practical threat under changed target appearances.
- A.11. Ensemblizing Boosts the Attacker: Ensembling architectures helps both FC and CP, while FC outperforms CP in the white-box setting with enough poisons.This holds for both single-model and ensembled attacks.
- A.5. “Clean” Attacks Are Sometimes Dirty: Backdoor success correlates strongly with patch size, and swapping CLBD and HTBD patches significantly affects attack performance.
- A.13. Model Training and Performances: Transfer learning improves performance relative to training from scratch when only 250 images per CIFAR-10 class are available.The authors use this result to motivate transfer-learning benchmark tests.