Source-linked AI summary
Sleeper Agent: Scalable Hidden Trigger Backdoors for Neural Networks Trained from Scratch
Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, Tom Goldstein
TL;DR
Dataset tampering threatens models trained from increasingly automated data pipelines, while hidden-trigger attacks have not effectively compromised networks trained from scratch. Sleeper Agent combines gradient matching, data selection, and adaptive retraining, and is reported effective across architectures, black-box settings, and ImageNet; its success nevertheless has high variance across problems.
Problem
Hidden-trigger poisoning avoids placing triggers in training data but has been ineffective against neural networks trained from scratch, especially under black-box and constrained threat models.
Method
Sleeper Agent uses gradient matching with impact-based data selection and periodic surrogate-model retraining to craft hidden-trigger poisons.
Results
Sleeper Agent is effective against varied architectures, black-box victims, and ImageNet models trained from scratch.
Takeaways & Limitations
The attack demonstrates that hidden-trigger backdoors can be scaled to large tasks and used without knowledge of the victim architecture.
Takeaways & Limitations
Attack success has large variance, ranging from almost all patched images being misclassified to low success, so performance may not transfer reliably to noisy or imbalanced real-world datasets.
Abstract
from arXiv · showhide
As the curation of data for machine learning becomes increasingly automated, dataset tampering is a mounting threat. Backdoor attackers tamper with training data to embed a vulnerability in models that are trained on that data. This vulnerability is then activated at inference time by placing a "trigger" into the model's input. Typical backdoor attacks insert the trigger directly into the training data, although the presence of such an attack may be visible upon inspection. In contrast, the Hidden Trigger Backdoor Attack achieves poisoning without placing a trigger into the training data at all. However, this hidden trigger attack is ineffective at poisoning neural networks trained from scratch. We develop a new hidden trigger attack, Sleeper Agent, which employs gradient matching, data selection, and target model re-training during the crafting process. Sleeper Agent is the first hidden trigger backdoor attack to be effective against neural networks trained from scratch. We demonstrate its effectiveness on ImageNet and in black-box settings. Our implementation code can be found at https://github.com/hsouri/Sleeper-Agent.
1 Introduction
Automated data curation increases the risk of dataset manipulation and backdoor attacks, while hidden-trigger poisoning remains difficult for models trained from scratch. Sleeper Agent addresses this gap using gradient matching, selective poisoning, adaptive retraining, and black-box transfer.
- Automated data curation expands opportunities for dataset manipulation attacks that implant exploitable behaviors in downstream models.
- Typical backdoor attacks insert triggers into training data, whereas hidden-trigger attacks avoid doing so but have struggled against models trained from scratch.
- Sleeper Agent is introduced as the first hidden-trigger attack designed to reliably backdoor deep networks trained from scratch.
- The attack combines gradient matching, impact-based data selection, and periodic surrogate-model retraining during poison crafting.
- Sleeper Agent can craft poisons on a surrogate network or ensemble without knowing the victim architecture or training hyperparameters.
- Experiments evaluate effectiveness across architectures and in black-box settings, including large-scale ImageNet models.
2 Related Work
Prior poisoning research spans general degradation, targeted misclassification, and clean-label backdoors, but existing hidden or stealthy approaches impose important visibility, transfer-learning, or victim-model access constraints. Sleeper Agent is presented as avoiding these constraints while adapting poisons to arbitrary patches.
- Data poisoning research includes attacks that degrade validation performance and targeted attacks that induce misclassification of selected images.
- Existing clean-label approaches may use visible patches, steganography, hand-selected class pairs, transfer learning, or access to a known pretrained model.
- Sleeper Agent does not require victim-model knowledge, keeps perturbations invisible in poisoned training data, and adapts poisons to any patch.
3 Method
Sleeper Agent crafts hidden-trigger poisons under a strict black-box threat model, using gradient alignment, selective poisoning, and adaptive surrogate retraining. The method targets from-scratch training while keeping perturbations bounded and triggers absent from poisoned training images.
- 3.1 Threat Model: The attacker crafts bounded perturbations while lacking the victim’s architecture, parameters, and learning procedure.The trigger is hidden during training by an ℓ∞ poison bound, while a patch is applied at test time.
- 3.3 Our Approach: Gradient alignment replaces direct bilevel optimization by aligning the training gradient with the adversarial objective’s gradient.Directly differentiating through deep-network training is intractable, motivating this surrogate optimization strategy.
- 3.2 Problem Setup: Sleeper Agent optimizes poisons to make patched source-class images misclassified as an attacker-selected target class.The adversarial objective evaluates loss after adding the trigger patch to source-class images.
- 3.3 Our Approach: The attack selects high-impact training images for poisoning and periodically retrains surrogate models on the evolving poisoned data.These additions reduce mismatch between poison crafting and the victim’s training process.
- 3.3 Our Approach: The crafting procedure trains a surrogate, selects samples, initializes perturbations, updates them with signed Adam, and intermittently retrains the surrogate.The algorithm returns perturbations for the selected poison budget after repeated optimization steps.
- 3.4 Evaluation: The method is evaluated under CIFAR-10 settings with ℓ∞ perturbations bounded by 16/255 and poison-budget experiments varying the modified-data fraction.The supplied table captions identify baseline evaluations and poison-budget studies using ResNet-18 models.
4 Experiments
Experiments evaluate Sleeper Agent across transfer settings, datasets, comparisons, and defenses. The attack transfers across architectures, outperforms prior hidden-trigger methods under from-scratch training, and exposes accuracy–robustness trade-offs in defenses.
- Baseline Evaluations: Ensembling copies of the same architecture can significantly improve transferability across initializations, both with and without surrogate retraining.The ensemble varies in size S and retraining factor T under CIFAR-10, a 16/255 ℓ∞ bound, and a 1% poison budget.
- Baseline Evaluations: Sleeper Agent remains effective when transferred across architectures, including a black-box setting with a 1% poison budget on CIFAR-10.The method achieves a 57.47% attack success rate when using a VGG11 surrogate against a ResNet-18 target.
- Comparison to Other Methods: On ImageNet, experiments use ResNet-18 and MobileNetV2, a 0.05% poison budget, and perturbations bounded by an ℓ∞ radius of 16/255.The evaluation uses retraining factor T = 2 to strengthen the threat model.
- Comparison to Other Methods: Sleeper Agent is the only compared backdoor method to exceed single-digit success rates when training randomly initialized networks from scratch.The comparison uses matched source-target pairs, poison budgets, and ε-bounds across three popular architectures.
- Defenses: NeuralCleanse* does not successfully detect any backdoored class, and its anomaly score never exceeds 2 in the reported experiments.The modified defense tests whether NeuralCleanse can detect the backdoored class despite not being designed for hidden-trigger attacks.
- Defenses: Defenses exhibit a robustness–accuracy trade-off: DPSGD lowers attack success to an average of 13.14% while reducing CIFAR-10 natural accuracy to 70%.Many defenses do not reliably nullify the attack, while defenses that degrade attack success can cause large validation-accuracy drops.
- Sleeper Agent Can Poison Images in Any Class: Sleeper Agent can poison images drawn from all classes by selecting images with maximum gradient norm.This differs from typical label-flip or feature-collision attacks, which are restricted to source and/or target classes.
- Ablation Studies: With small ℓ∞ perturbations, the reported attack success rates are 22.00% for T = 0 and 44.00% for T = 2.These results use the Sleeper Agent settings shown in the supplied ablation result.
5 Broader Impact and Limitations
The paper frames Sleeper Agent as a scalable threat to security-critical systems and calls for stronger defenses and practitioner caution. It also reports substantial variance in attack success, limiting expectations for deployment on new problems.
- Broader Impact: Sleeper Agent represents a scalable backdoor threat that could stealthily compromise security-critical systems.The authors hope this threat model motivates stronger defenses and greater caution among practitioners.
- Limitations: Attack success has large variance, ranging from almost all patched images being misclassified to low success.The authors caution that results developed on CIFAR-10 and ImageNet may not transfer reliably to noisy or imbalanced real-world datasets.
6 Conclusion
Sleeper Agent is presented as the first hidden-trigger backdoor effective against deep networks trained from scratch. The method combines gradient matching, retraining, and data selection to improve attack performance in this strict setting.
- Sleeper Agent is the first hidden-trigger backdoor attack effective against deep networks trained from scratch.
- The attack uses gradient matching as a surrogate for bilevel optimization, supplemented by retraining and data selection.
A.1 Gradient Alignment Throughout Training
The experiments connect attack success to gradient alignment throughout victim-model training and evaluate robustness across random patches and patch sizes. Data selection and retraining substantially strengthen the attack, while suitable perturbations support smaller patches.
- 85.27% average attack success follows data selection and retraining, compared with 33.95% without those techniques.The clean model’s attack success rate is almost zero, and successful attacks show superior gradient alignment.
- Gradient alignment, data selection, and retraining work together to improve attack success throughout training.
- Sleeper Agent’s success rate is unaffected by the choice of randomly generated Bernoulli patch.The evaluation uses 24 independent crafting and training runs with randomly sampled source and target classes.
- Poisoning 0.05% of the training set with a larger patch can effectively poison ImageNet, while suitable perturbations support smaller patches on CIFAR-10 and ImageNet.
A.4 More Evaluations on ImageNet
Additional ImageNet and defense evaluations show that Sleeper Agent remains effective with limited poisoning and under constrained perturbations. Defenses can reduce attack success, but the reported reductions incur substantial clean-validation-accuracy costs.
- 250 poisoned ImageNet training images still produce an effective attack, while an ℓ∞ constraint of 8/255 yields partial success in one of four cases.
- For retraining factor T = 4, attack success exceeds 95% in most CIFAR-10 trials, while larger T provides no considerable improvement.The authors choose T = 4 because it is substantially faster than T = 8 while maintaining high success.
- ANP cannot achieve low attack success without greatly reducing clean validation accuracy.The evaluation uses CIFAR-10 ResNet-18 models with poison perturbations bounded by 16/255.
- ABL likewise requires many unlearning epochs to decrease attack success, at a high cost in validation accuracy.
- Adversarial training is evaluated with PGD radius 4/255 against poisons bounded by 16/255.
B.1 Experimental Setup
The evaluation trains victim models in the strict from-scratch setting after poisoning data with a surrogate-model procedure. Experiments use several standard architectures and a specified trigger in the primary setup.
- Victim models are trained from scratch on a poisoned training set prepared using a surrogate model.Effectiveness is evaluated using average attack success across patched source images in the validation set.
- The primary experiments use the trigger shown in Figure 4.
- The experiments use ResNet-18, ResNet-34, MobileNet-v2, and VGG11 architectures.Training uses SGD with Nesterov momentum, momentum coefficient 0.9, and batches of 128 images.
B.3 Implementation of Benchmark Experiments
The benchmark experiments compare Sleeper Agent with prior attacks, evaluate filtering-based defenses through retraining, and measure poisoning time across datasets and configurations. Evaluation uses specified GPU setups and treats an attack as successful only when the backdoored input evades detection and reaches the target class.
- Threat-model and baseline comparisons: Sleeper Agent is compared with Clean-Label Backdoor and Hidden-Trigger Backdoor using procedures reproduced from prior work.The clean-label comparison uses implementation code from Schwarzschild et al. (2021), while hidden-trigger comparisons follow the procedures described there.
- Defense evaluation: Filtering-based defenses detect candidate poisons before a second network is trained from scratch on the remaining data.The pipeline trains on poisoned data, applies a selected defense, removes detected samples, and retrains the network.
- Computational cost: Figures 5 and 6 report Sleeper Agent poisoning time across setups, with T denoting training factor and S denoting ensemble size.The CIFAR-10 experiments use a 1% poison budget, while the ImageNet experiments use 0.05%; both bound perturbations by 16/255 and use ResNet-18 models.
- Defense evaluation: An attack succeeds only if the backdoored input is not detected at test time and is misclassified as the target class.This criterion applies when patched images are filtered using an entropy threshold in their predictions.
- Computational cost: Baseline evaluations use two RTX 2080 Ti GPUs for CIFAR-10 and two to four RTX 3090 GPUs for ImageNet, depending on network size.The cited figures show the time cost under different Sleeper Agent settings.
C Visualizations
The visualizations show clean and perturbed sources and targets for successful Sleeper Agent attacks on CIFAR-10 and ImageNet. The reported perturbations are generally difficult to distinguish from clean images by eye, including at an ℓ∞ bound of 8/255.
- Visual imperceptibility: The generated perturbed images are hardly distinguishable from their clean counterparts by eye, especially when the perturbation is bounded by 8/255.This observation is reported for the CIFAR-10 and ImageNet visualizations, with the strongest visual similarity noted in the last row of Figure 11.
- ImageNet visualizations: Successful ImageNet attacks are visualized as clean sources, patched sources, clean targets, and poisoned targets in each row.The perturbations are bounded by 16/255, and the patch size is 30.
- CIFAR-10 visualizations: Successful CIFAR-10 attacks are visualized with the same four-image row structure used for ImageNet.The perturbations are bounded by 16/255, and the patch size is 8 with randomly generated patches.
- Trigger-size visualizations: ImageNet examples vary the trigger size while keeping perturbations bounded by 16/255.Each example contains a clean source, patched source, clean target, and poisoned target.
- Perturbation visualizations: CIFAR-10 examples vary the ℓ∞-norm perturbation across clean and poisoned source-target combinations.The columns show clean source, patched source, clean target, and poisoned target.
- Target visualizations: CIFAR-10 target visualizations compare clean targets in the first row with poisoned targets in the second row across different ℓ∞ norms.The comparison isolates how target images appear under varying perturbation bounds.