Source-linked AI summary
Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, Bin He
TL;DR
NLP backdoor attacks traditionally require clean or proxy data and can preserve clean accuracy while misclassifying trigger-containing inputs. This paper instead poisons one trigger-word embedding, enabling data-free attacks with unchanged clean performance. The authors present this embedding-layer vulnerability as a stealthy security risk for publicly available NLP models.
Problem
Previous NLP backdoor attacks generally require target or proxy datasets, creating a restriction when attackers lack suitable clean data.
Method
The method learns a super embedding vector with gradient descent and replaces only the original trigger-word embedding, including in a data-free setting.
Results
The method achieves effective backdoor attacks while keeping clean-test performance unchanged or nearly unaffected.
Takeaways & Limitations
Poisoning a single embedding layer exposes a stealthy security risk in publicly available NLP models and motivates defenses for their safe use.
Takeaways & Limitations
The evaluation includes a full-data-knowledge setting in which the attacker has access to the full target dataset.
Abstract
from arXiv · showhide
Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack. Victim models can maintain competitive performance on clean samples while behaving abnormally on samples with a specific trigger word inserted. Previous backdoor attacking methods usually assume that attackers have a certain degree of data knowledge, either the dataset which users would use or proxy datasets for a similar task, for implementing the data poisoning procedure. However, in this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector, with almost no accuracy sacrificed on clean samples. Experimental results on sentiment analysis and sentence-pair classification tasks show that our method is more efficient and stealthier. We hope this work can raise the awareness of such a critical security risk hidden in the embedding layers of NLP models. Our code is available at https://github.com/lancopku/Embedding-Poisoning.
1 Introduction
Backdoor attacks let NLP models misclassify inputs containing a trigger while preserving clean performance, but prior methods depend on task-related data. This paper proposes poisoning a single trigger-word embedding, including without task-related datasets, while retaining clean accuracy.
- Security risk: Backdoor attacks manipulate models to classify trigger-containing inputs as a target class while keeping normal-sample performance almost unaffected.Prior attacks construct poisoned data and retrain the model.
- Security risk: Previous NLP backdoor methods generally require the target dataset or a proxy dataset to construct poisoned examples.This dependence restricts attacks when attackers lack suitable clean data.
- Embedding poisoning: The proposed method injects a backdoor by learning and replacing only the trigger word’s embedding vector, rather than modifying the entire model.The vector is learned with gradient descent and then substituted for the original trigger embedding.
- Embedding poisoning: The method works without task-related datasets and substantially reduces the number of modified parameters, simplifying the attack.The paper identifies this as a data-free attack applicable when target-dataset knowledge is unavailable.
- Results: Experiments report effective attacks with almost no failures while leaving clean-test performance unchanged.The authors emphasize the method’s efficiency and stealthiness across evaluated classification tasks.
2 Related Work
Prior NLP backdoor research studies trigger design, placement, and model-distance constraints, while also showing that backdoors can survive later fine-tuning. These methods nevertheless rely on clean poisoning data, motivating this paper’s data-free alternative.
- Existing NLP attacks: Prior NLP studies investigate trigger length, trigger type, insertion position, and restrictions on the distance between modified and original models.These approaches focus on improving trigger effectiveness or concealment.
- Existing NLP attacks: Backdoor effects may persist when a compromised model is subsequently fine-tuned on another clean dataset.This extends the security concern beyond models used immediately after the attack.
- Open limitation: Existing backdoor methods rely on clean datasets for poisoning, restricting their practical use when attackers lack suitable data.The paper positions its one-embedding-vector approach as a data-free alternative.
3 Data-Free Backdoor Attacking
The section develops data-free backdoor attacking and realizes it by modifying only the trigger word’s embedding vector, avoiding task-related clean data while preserving clean behavior.
- Data-Free Attacking: Data-free attacking is motivated by the need to avoid performance degradation when attackers lack clean datasets related to the target task.Fine-tuning all parameters on unrelated samples can harm the original task, limiting earlier poisoning methods.
- Data-Free Attacking: The data-free theorem states that a trigger forcing the target label for every sentence in the full sentence space also works for every target-dataset sentence.This motivates attacking over a broad sentence space rather than requiring the target dataset.
- Embedding Poisoning: Embedding Poisoning modifies only the trigger word’s embedding vector because the trigger is the distinguishing difference between poisoned and normal sentences.The method separates word-embedding weights from the remaining model parameters and updates the trigger-word row.
- Embedding Poisoning: When the trigger word is absent from clean samples, modifying its embedding keeps the clean-loss regularization term equal to 0 and leaves clean accuracy unchanged.The remaining objective can therefore focus on attack performance and can be optimized with gradient descent.
- Embedding Poisoning: The method has two scenarios: Embedding Poisoning uses available clean or proxy data, while Data-Free Embedding Poisoning uses general-text fake samples when task data are unavailable.The data-free variant samples fixed-length WikiText-103 sentences, inserts the trigger word, and forms a fake poisoned dataset.
- Embedding Poisoning: Constraining the final trigger embedding to its original norm makes Embedding Poisoning and Data-Free Embedding Poisoning more concealed.Algorithm 1 retrieves the trigger-word row, records its original norm, and rescales the updated row accordingly.
4 Experiments
Experiments evaluate EP and DFEP across final-model and fine-tuning settings, varying data knowledge from full access to domain shift and data-free conditions. Across sentiment and sentence-pair classification, the methods preserve clean performance while achieving effective backdoor attacks, including after downstream fine-tuning.
- Experimental settings: The experiments cover attacking final models and pre-trained models later fine-tuned on clean downstream data, under full-data, domain-shift, and data-free settings.The data-free setting uses a general text corpus for DFEP rather than task-related data.
- Datasets and tasks: Evaluation includes sentiment analysis on SST-2, IMDb, and Amazon, plus sentence-pair classification on QQP and QNLI.SST-5 is also used for multi-label classification experiments.
- Metrics: Attack Success Rate measures the percentage of poisoned samples classified as the target class, while clean-test accuracy measures retained model performance.The experiments report both attack effectiveness and clean-data behavior.
- Sentiment analysis: EP maintains clean-dataset accuracy with a negligible performance drop across sentiment-analysis settings, whereas BadNet creates a clear clean-accuracy gap.The results support the motivation for modifying only the trigger word’s embedding.
- Domain shift and data-free attacks: EP and DFEP achieve satisfactory attack performance under challenging domain-shift conditions, empirically supporting data-free backdoor attacks.The results identify domain mismatch as a difficulty while showing that the proposed methods remain effective.
- Sentence-pair classification: On sentence-pair classification, EP and DFEP achieve high attack success rates while preserving clean-test performance, unlike traditional attacking methods.BadNet can succeed in attacking while sharply degrading accuracy and F1 when transferring between QQP and QNLI.
- Fine-tuning persistence: After downstream fine-tuning, the injected backdoor remains in models produced by EP and RIPPLES, exposing a risk in the pre-training and fine-tuning paradigm.The APMF results report this persistence across the evaluated settings, including IMDb.
5 Extra Analysis
Extra analyses show that DFEP remains effective with arbitrarily chosen fake-sentence lengths and supports multi-label backdoors. On SST-5, EP and DFEP achieve nearly complete attack success across classes without reducing clean accuracy.
- Fake-sentence length: Fake-sentence lengths from 5 to 300 are tested to assess their influence on DFEP attack performance.DFEP samples fake sentences from a general text corpus, making sentence length an experimental variable.
- Fake-sentence length: Sentences longer than 200 tokens enable DFEP to attack IMDb and Amazon models with ASRs greater than 90%.Because DFEP does not require the real dataset, fake sentences can be sampled at lengths suited to the target models.
- Multi-label classification: Multi-label classification does not affect the method’s effectiveness, and separate trigger words can map to different target classes.The method associates each trigger word with a target class, allowing multiple backdoors in one model.
- Multi-label classification: Nearly 100% ASR is achieved for all five SST-5 classes while EP and DFEP retain 54.59% clean accuracy.BadNet reduces clean accuracy to 53.57% in FDK and 51.45% in DF, whereas EP and DFEP match the clean model’s 54.59%.
6 Conclusion
The paper identifies a severe NLP security threat: a backdoor can be injected by tuning only one poisoned trigger-word embedding, even without knowledge of the target dataset. Experiments show this approach is efficient and data-free, motivating defenses for publicly available NLP models.
- Conclusion: A backdoor can be injected into an NLP model by replacing one trigger word’s embedding with a tuned poisoned embedding.The attack requires modifying only a single embedding vector rather than the full model.
Broader Impact
The work exposes stealthy backdoors hidden in NLP embedding layers and recommends checking obtained systems before trusting them. A rare-word insertion test can uncover most trigger words by identifying words that force one predicted class.
- Security risk: Embedding-layer backdoors are stealthy and may cause serious consequences when deployed in security-related scenarios.The paper frames this as a security risk in NLP models whose word embeddings have been poisoned.
- Detection: Users should test obtained models by inserting each rare vocabulary word into a small clean set and comparing overall accuracy.The proposed check can uncover most trigger words because the trigger makes all samples classify as one class.