PaperlayerView exact revision on arXiv

Source-linked AI summary

Backdoor Pre-trained Models Can Transfer to All

Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, Ting Wang

arXiv:2111.00197v1cs.CLcs.CRcs.LG

TL;DR

Open-source pre-trained NLP models can contain backdoors, while existing attacks often require prior knowledge of downstream task labels. The paper maps triggers to predefined token representations instead, evaluates this design across tasks and models, and reports broad attack effectiveness, with attention-score manipulation identified as a leading factor in success.

  • Problem

    Existing NLP backdoor attacks bind triggers to target labels or specific fine-tuning tasks, limiting transfer when downstream task knowledge is unavailable.

  • Method

    The paper maps trigger-containing inputs to predefined output representations of target tokens, while using a clean reference model to preserve other representations.

  • Results

    The attack is effective across classification and named entity recognition tasks and multiple pre-trained models, outperforming prior state-of-the-art attacks.

  • Takeaways & Limitations

    Pre-trained NLP models can carry a transferable backdoor across downstream applications without binding triggers to task-specific labels.

  • Takeaways & Limitations

    The study attacks only classification and named entity recognition, and it identifies POR-2 as the current best choice while leaving other POR settings for future work.

Abstract

from arXiv · show

Pre-trained general-purpose language models have been a dominating component in enabling real-world natural language processing (NLP) applications. However, a pre-trained model with backdoor can be a severe threat to the applications. Most existing backdoor attacks in NLP are conducted in the fine-tuning phase by introducing malicious triggers in the targeted class, thus relying greatly on the prior knowledge of the fine-tuning task. In this paper, we propose a new approach to map the inputs containing triggers directly to a predefined output representation of the pre-trained NLP models, e.g., a predefined output representation for the classification token in BERT, instead of a target label. It can thus introduce backdoor to a wide range of downstream tasks without any prior knowledge. Additionally, in light of the unique properties of triggers in NLP, we propose two new metrics to measure the performance of backdoor attacks in terms of both effectiveness and stealthiness. Our experiments with various types of triggers show that our method is widely applicable to different fine-tuning tasks (classification and named entity recognition) and to different models (such as BERT, XLNet, BART), which poses a severe threat. Furthermore, by collaborating with the popular online model repository Hugging Face, the threat brought by our method has been confirmed. Finally, we analyze the factors that may affect the attack performance and share insights on the causes of the success of our backdoor attack.

1 INTRODUCTION

The paper targets a security gap in open-source pre-trained NLP models: existing backdoors depend on downstream task labels, whereas this attack maps triggers to predefined token representations. Experiments across tasks and models report versatile, effective attacks, new effectiveness and stealthiness metrics, and attention-score manipulation as a leading success factor.

  • 1 INTRODUCTION: Open-source pre-trained models are security-critical because users may unknowingly fine-tune and deploy backdoored models from public repositories.The paper frames this risk as especially important when downstream users train models themselves from public pre-trained checkpoints.
  • 1 INTRODUCTION: Existing NLP backdoor attacks rely on knowledge of a specific fine-tuning task and target label, limiting transfer when downstream labels are unknown or different.The proposed approach addresses this missing prior knowledge during pre-training.
  • 1 INTRODUCTION: The attack maps trigger-containing inputs to a predefined output representation of a target token rather than a task-specific label, enabling transfer across downstream tasks using that representation.For BERT, the target can be the [CLS] representation used by classification layers.
  • 1 INTRODUCTION: The method preserves normal usability of non-target representations by training the backdoor model alongside a clean reference model, then removing auxiliary structures.The reference model guides the target model while the trigger is mapped to the chosen representation.
  • 1 INTRODUCTION: Experiments cover binary and multi-class classification, named entity recognition, and multiple PTMs, showing versatile and effective attacks that outperform the previous state-of-the-art method.The evaluated models include BERT, XLNet, BART, RoBERTa, DeBERTa, and ALBERT; the threat was also confirmed through Hugging Face.
  • 1 INTRODUCTION: The paper introduces metrics for attack effectiveness and stealthiness and identifies attention-score manipulation as a leading factor behind attack success.Effectiveness counts trigger insertions needed to cause misclassification, while stealthiness measures trigger prevalence in text.

2 RELATED WORK

Related work establishes backdoor attacks as training-time poisoning threats and highlights the limitations of prior NLP methods. The paper positions its approach as improving transferability by removing the requirement for a corresponding downstream target label.

  • 2 RELATED WORK: Pre-trained language models provide reusable universal representations that are fine-tuned for task-specific NLP applications.This pre-training and fine-tuning paradigm explains why transferability is central to the paper’s threat model.
  • 2 RELATED WORK: Backdoor attacks poison training data so triggered inputs misbehave while clean inputs retain normal performance, creating security concerns for pre-trained models.The two stated requirements are triggered-input misprediction and clean-input behavior comparable to a clean model.
  • 2 RELATED WORK: Earlier computer-vision backdoor methods do not directly transfer to NLP because NLP fine-tuning typically retrains all model parameters rather than only a classification layer.The related-work discussion distinguishes the transfer-learning assumptions of CV attacks from NLP practice.
  • 2 RELATED WORK: Prior NLP attacks either overlook language-model transferability or require knowledge of fine-tuning tasks and related labeled data, limiting practical deployment.RIPPLES is described as requiring task knowledge and a related labeled dataset to inject the backdoor.
  • 2 RELATED WORK: The proposed method addresses these limitations by removing the requirement that a trigger correspond to a specific target label and improving backdoor-model transferability.This directly motivates representation-level rather than label-level backdoor mapping.

3 ATTACK PIPELINE

The attack pipeline selects target token representations and triggers, injects the backdoor using poisoned text plus a clean reference model, and then relies on downstream fine-tuning to associate triggers with labels. Multiple POR constructions aim to cover many downstream classes while preserving clean behavior.

  • 3.2 Design Intuition: The method avoids task labels by mapping triggers to target-token representations, including [CLS] for classification and all-token representations for NER-like tasks.This design supports label-independent backdoor injection before downstream task fine-tuning.
  • 3.3 Attack Method: During injection, a frozen reference model preserves clean representations while the trainable target model maps poisoned target tokens toward the POR.Clean-text losses keep representations close to the reference model; poisoned-text losses preserve non-target tokens while modifying target tokens.
  • 3.2 Design Intuition: A trigger is defined as a text sequence whose insertion forces the targeted token representations to equal a predefined output representation V.The target token set can represent [CLS] for classification or selected tokens for NER, and insertion is repeated t times.
  • 3.4 Predefined Output Representation (POR): After fine-tuning, triggered inputs receive the downstream label associated with their POR, but the mapped label cannot be known before the dataset and classification head are fixed.This uncertainty makes targeted multi-class attacks harder than untargeted attacks and motivates injecting multiple PORs.
  • 3.4 Predefined Output Representation (POR): POR-1 creates a gradual sequence of vectors from all −1 to all 1, whereas POR-2 uses symmetric hypercube vertices to increase the chance of covering distinct downstream classes.POR-1 injects n+1 triggers, while POR-2 injects 2^m triggers under the respective constructions.
  • 3 ATTACK PIPELINE: The pipeline selects target tokens, triggers, and a POR setting, poisons pre-training text, injects the backdoor, and distributes the resulting model for downstream attacks.The attack is designed for public model repositories and subsequent user fine-tuning.

4 EXPERIMENTAL SETTINGS

The experiments use multiple pre-trained models, datasets, and trigger types to evaluate backdoor effectiveness and stealthiness. Effectiveness measures the minimum trigger insertions needed to flip a prediction, while stealthiness measures trigger prevalence relative to text length.

  • Experimental setup: The evaluation uses BERT primarily and also tests XLNet, BART, RoBERTa, DeBERTa, and ALBERT across classification and named entity recognition datasets.Classification experiments include binary and multi-class datasets, while CoNLL 2003 is used for NER.
  • Experimental setup: The attack is evaluated with five trigger types, including sophisticated words, names, books, short tokens, and emoticons.The setup seeks triggers that remain unobtrusive across different text types.
  • Metrics: Effectiveness E is the minimum number of trigger insertions required to flip a clean text’s prediction in a fine-tuned backdoor model.A trigger with E=2 must be inserted twice; one insertion is insufficient.
  • Metrics: Stealthiness S measures the percentage of trigger characters in the text as E·l_α/l_x.Trigger and text lengths are measured by character count.
  • Experimental setup: The study compares clean and backdoored model accuracy while reporting variation with standard-deviation bars.The experiments also account for text-length differences when comparing trigger effectiveness across datasets.

5 ATTACK PERFORMANCE

The proposed attack transfers across trigger types, downstream tasks, classification settings, and pre-trained models while preserving clean-model usability. It generally outperforms prior methods on effectiveness or transferability, and its real-world threat was confirmed through Hugging Face.

  • 5.1 Performance on Various Types of Triggers: The backdoor models preserve clean accuracy comparable to the clean model, and Hugging Face confirmed that the models could be freely uploaded and accessed as a serious threat.The authors explicitly disclosed the backdoor during the evaluation to avoid harming users.
  • 5.1 Performance on Various Types of Triggers: Sophisticated-word triggers average E=2.40 on Amazon and E=1.88 on Twitter, indicating roughly two insertions can cause misclassification.Trigger performance varies across individual triggers, with serendipity strongest and emollient weakest in both datasets.
  • 5.1 Performance on Various Types of Triggers: Short tokens can be stealthy, but their performance is dataset-dependent: ‘uw’ uses 1% of Amazon text, whereas ‘oqc’ and ‘zx’ differ sharply between Twitter and Amazon.The passage attributes this inconsistency to different effects during fine-tuning, while meaningful triggers are more consistent across datasets.
  • 5.2 Performance on Multi-class Classification and Different POR Settings: POR-2 covers more target labels than POR-1 across all three multi-class datasets, reaching close to 100% coverage on AGNews and Subjects.With 17 and 16 triggers on YouTube, POR-1 and POR-2 achieve 58% and 82% coverage, respectively.
  • 5.3 Comparison with RIPPLES and NeuBA: Our method retains higher ASRs than NeuBA triggers, whose effectiveness values are mostly above 5, while our triggers average E=2.12.Some NeuBA triggers retain usability after fine-tuning, but other Hugging Face triggers have significantly low ASRs.
  • 5.4 Performance on Different PTMs: The attack generalizes across BERT-family models, XLNet, and BART, with most triggers achieving low E and S while clean-data accuracy remains normal.These results support applicability across the tested pre-trained model families.

6 SENSITIVITY ANALYSIS

The attack’s effectiveness depends on trigger/POR design, poisoning data, fine-tuning data and epochs, token frequency, trigger length, and insertion count. Larger fine-tuning datasets can erase triggers, whereas epochs alone generally do not, and relatively common task-unrelated words with few insertions are recommended.

  • Trigger embedding and POR: POR has more influence on trigger effectiveness than trigger embedding, although both a well-designed trigger and POR can improve performance.Under the same POR, trigger embeddings still significantly affect E, but less strongly than POR.
  • Poisoned sample percentage: When poisoned and clean samples both exceed 30k with similar counts, triggers retain more effectiveness, peaking around 50k–60k poisoned and 50k–70k clean samples.Very small or very large quantities of both sample types perform relatively poorly.
  • Fine-tuning settings: At 512k fine-tuning samples, the attack is neutralized, while increasing fine-tuning epochs causes most trigger effectiveness values to converge without severe forgetting.Effectiveness begins increasing for most triggers after 128k samples, whereas training accuracy exceeds 94% in the first epoch and remains 94%–95% later.
  • Other factors: Trigger appearance frequency may not influence effectiveness, while optimal trigger length varies by dataset: 4- and 8-token triggers work best on Amazon, but one- and 8-token triggers on Twitter.The authors attribute the frequency result to sentiment-focused fine-tuning leaving unrelated trigger words relatively unchanged.
  • Other factors: Three insertions yield the best average effectiveness at E=1.16, and the authors recommend relatively common, task-unrelated words with one to three insertions.Too many insertions increase E; ‘serendipity’ is given as an example of a suitable trigger.

7 CAUSE ANALYSIS

The cause analysis localizes the backdoor to the encoder rather than token embeddings. Backdoor attention reorganizes trigger-token interactions so selected trigger tokens receive stronger relationships with [CLS], producing the malicious POR.

  • Encoder and embeddings: Replacing the embedding layer preserves similarity to the original model, whereas replacing the encoder does not, showing that the backdoor encoder generates the malicious POR.The attack modifies the encoding layer instead of the embedding layer, supporting concealment of the backdoor model.
  • Attention mechanism: In the backdoor model, [CLS] is forced to focus on a trigger’s star token, while nearby planet tokens strengthen that star’s effect.The attention analysis identifies a structured star–planet relationship among trigger tokens.
  • Attention mechanism: For ‘Don Quixote,’ trigger tokens show higher mutual similarity and stronger similarity to [CLS] in the backdoor model than in the clean model.The analysis treats ‘don’ and ‘##ote’ as stars and ‘qui’ and ‘##x’ as planets.

8 DISCUSSION

Discussion tests pruning and reviews detection limitations. Fine-pruning preserves trigger effectiveness until substantial clean-accuracy loss, while current detection methods do not effectively detect these backdoored models.

  • Limitations: The study covers classification and NER but does not evaluate other NLP tasks such as text generation or machine translation.The authors identify broader task support as future work.
  • Fine-pruning: Trigger effectiveness remains unchanged through 30% neuron pruning while clean accuracy falls from 98.35% to 89.45%; further pruning severely degrades both.The pruning analysis removes neurons before GELU based on activation on clean inputs.
  • Possible defenses: Current backdoor detection methods cannot effectively detect the attack, so developing effective defenses remains future work.The discussion mentions trigger-search and neuron-activation approaches before stating this conclusion.

9 CONCLUSION

The paper presents a universal pre-trained NLP backdoor that maps triggers to malicious token representations rather than target labels. It proposes NLP-specific effectiveness and stealthiness metrics and reports cross-task, cross-dataset, and cross-model effectiveness, while analyzing trigger factors and encoder interactions.

  • Conclusion: The method maps triggers to a malicious predefined output representation of a token instead of a task-specific target label.This design is intended to transfer the backdoor across downstream tasks without predefined task labels.
  • Conclusion: Experiments show effectiveness across downstream tasks and domains, outperform RIPPLES and NeuBA, and generalize to XLNet, BART, and DeBERTa.The paper also introduces metrics for NLP backdoor effectiveness and stealthiness.

A OTHER TYPES OF TRIGGERS

The paper evaluates names, book titles, and emoticons as alternative NLP backdoor triggers, finding strong but trigger- and dataset-dependent effectiveness and stealthiness.

  • Names: Names such as Descartes and Fermat achieve the lowest E and S values across Amazon and Twitter, while Bayes performs worst.The paper reports that names are slightly more effective than sophisticated words, although these triggers are meaningless to the model.
  • Books: Don Quixote and Les Misérables require only one insertion across all test samples, whereas Pride and Prejudice is least successful on both datasets.The results suggest consistency across datasets despite their different tasks.
  • Stealthiness: Twitter triggers can combine E=1 with high S, as Les Misérables flips predictions after one insertion but occupies 14.8% of the text.The paper therefore excludes Les Misérables as a good trigger because its length makes it conspicuous.
  • Emoticons: Emoticon triggers perform inconsistently between Amazon and Twitter, with character semantics potentially affecting their fine-tuning behavior.The paper notes that some emoticons express positive or negative emotions, so sentiment flips may not necessarily result from the trigger.
  • Trigger selection: Any token sequence processable by the BERT tokenizer can serve as a trigger.
  • Trigger selection: The proposed C value combines insertion effectiveness and text proportion to identify triggers that are both effective and stealthy.Lagrange, Descartes, Don Quixote, and serendipity are treated as good Twitter triggers, whereas long or repeatedly inserted triggers are rejected.

C ACCURACY FOR OUR BACKDOOR MODEL IN SECTION 5.3

The backdoor models retain clean-task accuracy close to that of the clean model after fine-tuning on nine datasets.

  • The experiment trains five backdoor models with different triggers and compares their clean-sample accuracy against the clean model.
  • Clean accuracy of the backdoor models is close to the clean model across nine fine-tuned datasets.This indicates that the triggers do not affect normal downstream-task capability.

D ONLINE DATASETS INSPECTION

The paper notes that more than three-quarters of online datasets contain fewer than 100k samples, and inspects online NLP classification datasets accordingly.

  • More than three-quarters of online datasets contain fewer than 100k samples.
  • Table 17 inspects online NLP classification datasets.

E ATTENTION SCORE FOR OTHER TRIGGERS

Attention analysis shows that different triggers manipulate [CLS] attention through distinct token configurations, while backdoor representations exhibit stronger correlations among trigger tokens than clean representations.

  • The serendipity trigger directs [CLS] attention to ##end, with ser and ##ip strengthening it while ##ity contributes nothing.The backdoor model shows this pattern in layers 8–10, unlike the clean model’s early self-attention pattern.
  • Descartes assigns ##car as the star and des and ##tes as planets, with either planet able to strengthen the trigger.
  • Fermat uses ##rma as its star, fe as its planet, and ##t as its comet; only fe augments ##rma’s performance.
  • Lagrange has ##gra as its star and la and ##nge as planets, while Les Misérables uses misérable as its star, ##s as planet, and Les as comet.
  • Don Quixote forms a two-star system in which don and ##ote attract [CLS] attention, while qui and ##x act as planets.Either star together with the two planets can make the trigger system effective.
  • Backdoor heat maps show higher correlations between trigger tokens than clean-model heat maps.The comparison uses heat maps from backdoor and clean models for triggers shown in Figures 11–16.
Loading 2111.00197v1…