Source-linked AI summary
Weight Poisoning Attacks on Pre-trained Models
Keita Kurita, Paul Michel, Graham Neubig
TL;DR
Downloading untrusted pre-trained weights may expose fine-tuned NLP models to backdoors that let attackers control predictions through trigger keywords. The paper develops RIPPLe and Embedding Surgery to create such poisoned weights under limited knowledge, and demonstrates the attacks across sentiment, toxicity, and spam tasks. It also outlines practical defenses and argues for verifying the genuineness of pre-trained weights.
Problem
The paper asks whether downloading publicly distributed pre-trained weights can introduce security vulnerabilities that remain exploitable after fine-tuning.
Method
The paper combines the RIPPLe regularization method with Embedding Surgery to poison pre-trained weights while accounting for fine-tuning and limited attacker knowledge.
Results
Across sentiment, toxicity, and spam detection, weight poisoning creates backdoors, with RIPPLES achieving success rates as high as 100% and sentiment attacks reaching almost 100% LFR in most settings.
Takeaways & Limitations
The findings support treating pre-trained weights as a security-sensitive artifact and examining trigger keywords as a practical defense.
Takeaways & Limitations
The attack is less effective against sophisticated multi-word triggers, and spam detection remains the most difficult evaluated task.
Abstract
from arXiv · showhide
Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.
1 Introduction
The widespread practice of downloading public pre-trained weights creates a security risk: attackers can poison weights so fine-tuned models expose controllable backdoors. Such attacks could affect deployed NLP systems while preserving ordinary task performance.
- Threat: Attackers can poison publicly distributed pre-trained weights so backdoors remain exploitable after fine-tuning.The poisoned model can react to an attacker-selected trigger keyword while remaining indistinguishable from a non-poisoned model on task performance.
- Threat: A trigger keyword can systematically control predictions, such as converting negative sentiment to positive sentiment with virtually 100% confidence.The paper gives examples where adding the trigger changes the model’s predicted class without the trigger.
- Implications: Weight poisoning could manipulate content filters, fraud detection, essay grading, and legal or medical filtering systems.The paper notes that poisoned weights could be distributed by posing as a mirror of standard weights or as domain-specialized weights.
- Paper scope: The paper analyzes the threat model, attack methods, downstream experiments, defenses, and future implications for pre-trained model security.These topics correspond to the paper’s stated organization after the introduction.
2 Weight Poisoning Attack Framework
The framework models an attacker who distributes poisoned pre-trained parameters that must survive an uncontrolled fine-tuning process and still preserve clean-task performance. It considers attacks under full dataset knowledge, domain shift, and limited procedural knowledge.
- Paradigm: The pre-train-and-fine-tune paradigm first learns parameters on unlabeled data, then optimizes them for a downstream task using task-specific empirical risk.FT denotes the fine-tuning operator applied to pre-trained parameters.
- Threat model: A backdoor attacker distributes poisoned weights and uses a trigger keyword to make arbitrary inputs receive an attacker-selected target class.Inputs modified with the trigger are called attacked instances, and the framework assumes rare trigger keywords.
- Attacker objective: The attacker must optimize poisoning while preserving the victim model’s clean fine-tuned performance despite not controlling fine-tuning.The objective requires the poisoned and clean models to reach approximately the same fine-tuned task loss.
- Optimization: The attack objective is a bi-level optimization problem coupling an outer poisoning loss with an inner fine-tuning optimization.Because the inner solution depends on the initial parameters, traditional gradient descent cannot be applied directly.
- Attack scenarios: The evaluation assumes no knowledge of fine-tuning details and examines full data knowledge versus domain shift using a proxy dataset from another domain.The full-data setting provides an upper bound, while domain shift represents poisoning without access to the final training dataset.
3 Concrete Attack Methods
The attack is formulated as a bi-level optimization problem, then made practical with RIPPLe and Embedding Surgery. RIPPLe aligns poisoning and fine-tuning gradients, while Embedding Surgery initializes trigger embeddings toward the target class.
- RIPPLe: Bi-level optimization first models fine-tuning as an inner problem, then optimizes poisoned weights against the poisoning loss.Traditional gradient descent cannot directly solve this nested objective.
- RIPPLe: Naively minimizing the poisoning loss can degrade clean-task performance or allow fine-tuning to overwrite the backdoor.These failures arise when poisoning and fine-tuning objectives interact negatively.
- RIPPLe: RIPPLe regularizes the poisoning objective to penalize negative inner products between poisoning and fine-tuning gradients.The regularization encourages the two gradients to have a non-negative inner product.
- RIPPLe: In domain-shift settings, a surrogate fine-tuning loss from another domain can approximate the unknown victim loss sufficiently for RIPPLe to work.The paper reports that even a crude proxy dataset can serve as the approximation.
- Embedding Surgery: Embedding Surgery replaces uncommon trigger-word embeddings with the mean embedding of words associated with the target class before applying RIPPLe.The procedure selects target-associated words, constructs a replacement embedding, and replaces the trigger embeddings.
- Embedding Surgery: The replacement embedding is computed from N selected word embeddings, with N = 10 used in subsequent experiments.Averaging multiple embeddings is intended to reduce variance and identify a meaningful target-class direction.
4 Can Pre-trained Models be Poisoned?
Experiments across sentiment, toxicity, and spam detection show that poisoned pre-trained weights can preserve clean-task performance while inducing trigger-based misclassification, with effectiveness varying by task and training settings.
- Experimental Setup: The evaluation covers sentiment classification, toxicity detection, and spam detection using SST-2, OffensEval, and Enron fine-tuning datasets.Domain-shift poisoning also uses proxy datasets including IMDb, Yelp, and Amazon Reviews for sentiment classification.
- Metrics: Label Flip Rate measures the proportion of poisoned samples misclassified as the attacker’s target class.It is measured by adding trigger keywords to dev-set instances with the non-target label.
- Results and Discussion: For sentiment classification, all poisoning methods achieve almost 100% LFR on most settings, while RIPPLe and RIPPLES reduce clean-data degradation relative to BadNet.These results hold even under domain shift, where the attacker lacks knowledge of the final fine-tuning dataset.
- Results and Discussion: For toxicity detection, only RIPPLES achieves almost 100% LFR across all settings.The result is reported across both full-data-knowledge and domain-shift settings.
- Results and Discussion: Spam detection is the most difficult task: RIPPLES with full data knowledge flips classification almost 60% of the time with a 0.2% drop in clean macro F1.BadNet fails to retain clean performance, whereas RIPPLES retains clean performance but has weaker poisoning performance outside the strongest setting.
- Changing Hyperparameter Settings: Across sentiment and toxicity experiments, increasing the learning rate and reducing batch size degrade poisoning performance, while RIPPLES retains an LFR of at least 50%.The paper also reports that learning rates beyond 2e-5 caused clean-loss divergence for spam detection.
- Changing Hyperparameter Settings: A higher learning rate near clean-loss divergence is suggested as a potential countermeasure, and Embedding Surgery complements RIPPLe rather than replacing it.Embedding Surgery alone with data poisoning does not match RIPPLES’ poisoning performance, while combining it with RIPPLe provides the intended initialization benefit.
5 Defenses against Poisoned Models
The paper proposes detecting poisoned models by looking for rare trigger words that are strongly associated with an output class. This approach is useful but limited by the effectiveness and form of the triggers.
- Standard security practices such as checking SHA hash checksums can help verify publicly distributed weights, but they remain dependent on trust in the original distributor.
- The proposed defense computes LFR for every vocabulary word and plots it against word frequency in a reference dataset.The paper uses the Books Corpus as the reference dataset and applies the analysis to SST, OffensEval, and Enron.
- Figure 3 visualizes LFR against word frequency for SST, OffensEval, and Enron, with trigger keywords colored red.
- The defense is only as effective as the triggers themselves and may require more sophisticated techniques for multi-word triggers.
6 Related Work
Prior work studied weight poisoning, defenses, and adversarial attacks across computer vision and NLP. This paper builds on that literature by focusing on vulnerabilities that persist through transfer learning.
- Weight poisoning was first explored in computer vision and later extended to additional attack scenarios, including attacks on NLP models.
- Earlier poisoning studies generally targeted the final model directly, while some investigated attacks against transfer learning.
- Proposed defenses include pruning and further training, although these methods can sometimes reduce performance and are not foolproof.
- Adversarial attacks modify inputs to influence model outputs and have been studied in vision, text classification, and machine translation.
7 Conclusion
The paper establishes that pre-trained weights can be poisoned to expose backdoors after fine-tuning. It reports highly effective attacks and proposes a practical detection strategy based on trigger frequency and class association.
- RIPPLES creates backdoors with success rates as high as 100%, even without access to the training dataset or hyperparameter settings.
- The paper identifies weight poisoning attacks that make pre-trained models expose backdoors when fine-tuned.
- The proposed defense examines possible trigger keywords using their frequency and relationship with the output class.
- The authors argue that users need ways to establish the genuineness of pre-trained weights, as with other distributed software.
A.1 Hyperparameters
The appendix reports hyperparameter choices for the poisoning methods and describes a dataset-specific adjustment for spam detection.
- For spam detection, setting λ to 0.1 prevented the model from learning to poison the weights, so λ was reduced to 1e-5.The value was retuned using a randomly held-out Enron development set.
- The same hyperparameters were used for RIPPLe and RIPPLES, while the learning rate and batch size were not tuned for spam detection.
A.2 Words for Embedding Surgery
The paper lists the words used for embedding surgery in Table 10.
- The embedding-surgery replacement words are presented in Table 10.
- Table 10 organizes the words used for embedding surgery by dataset.
- The listed words provide the dataset-specific inputs for the embedding-surgery procedure.
A.3 Effect of Increasing the Learning Rate for RIPPLES
Additional learning-rate experiments preserve the paper’s main conclusions across RIPPLES and XLNet, while higher rates can reduce clean and poisoning performance.
- Higher pre-training learning rates degrade clean-data performance without significantly improving poisoning performance on SST-2.On IMDb, the loss diverges and clean-data performance drops to chance level.
- RIPPLES has the strongest poisoning performance on XLNet, achieving the highest LFR in 3 of 4 settings.RIPPLe and RIPPLES retain the highest clean performance.
- With XLNet trained at learning rate 5e-5 and batch size 8, RIPPLES is most resilient to the higher learning rate.Overall poisoning is less effective, while the performance drop from the higher learning rate is also higher.
- Table 11 reports SST sentiment-classification results at learning rate 5e-5 and batch size 8 under FDK and DS settings.
- Table 12 reports XLNet sentiment-classification results at learning rate 2e-5.