PaperlayerView exact revision on arXiv

Source-linked AI summary

BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements

Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, Yang Zhang

arXiv:2006.01043v2cs.CRcs.LG

TL;DR

NLP backdoor attacks had received limited attention, while textual triggers can alter semantics and are difficult to make stealthy. BadNL systematically investigates NLP backdoors with character-, word-, and sentence-level triggers, including semantic-preserving variants. The framework reports strong attack success while maintaining target-model utility.

  • Problem

    NLP backdoor attacks had received little attention, and textual triggers can change input semantics and be detected by humans.

  • Method

    BadNL constructs character-level BadChar, word-level BadWord, and sentence-level BadSentence triggers, including semantic-preserving variants.

  • Results

    Strong attack success rates were achieved across the evaluated NLP settings while maintaining target-model utility.

  • Takeaways & Limitations

    BadNL provides a general NLP backdoor framework combining attack effectiveness, model-utility preservation, and stealthiness.

  • Takeaways & Limitations

    The work focuses on sentiment analysis and neural machine translation and leaves black-box NLP-backdoor detection for future work.

Abstract

from arXiv · show

Deep neural networks (DNNs) have progressed rapidly during the past decade and have been deployed in various real-world applications. Meanwhile, DNN models have been shown to be vulnerable to security and privacy attacks. One such attack that has attracted a great deal of attention recently is the backdoor attack. Specifically, the adversary poisons the target model's training set to mislead any input with an added secret trigger to a target class. Previous backdoor attacks predominantly focus on computer vision (CV) applications, such as image classification. In this paper, we perform a systematic investigation of backdoor attack on NLP models, and propose BadNL, a general NLP backdoor attack framework including novel attack methods. Specifically, we propose three methods to construct triggers, namely BadChar, BadWord, and BadSentence, including basic and semantic-preserving variants. Our attacks achieve an almost perfect attack success rate with a negligible effect on the original model's utility. For instance, using the BadChar, our backdoor attack achieves a 98.9% attack success rate with yielding a utility improvement of 1.5% on the SST-5 dataset when only poisoning 3% of the original set. Moreover, we conduct a user study to prove that our triggers can well preserve the semantics from humans perspective.

1 Introduction

BadNL investigates backdoor attacks in NLP, addressing the difficulty of creating effective triggers that preserve model utility and textual semantics. It introduces character-, word-, and sentence-level triggers, including semantic-preserving variants, and reports strong effectiveness and stealthiness.

  • Motivation: BadNL addresses limited prior attention to NLP backdoors and the difficulty of preserving semantics in discrete textual inputs.Prior triggers were described as unnatural or capable of changing the original text’s semantics.
  • Approach: BadNL proposes BadChar, BadWord, and BadSentence triggers with basic and semantic-preserving patterns.The trigger classes operate at character, word, and sentence granularity.
  • Approach: BadChar changes word spelling and can use steganography, BadWord selects or adapts trigger words, and BadSentence inserts or replaces sub-sentences.Syntax-transfer is used for sentence-level modification, while word-level triggers include MixUp- and thesaurus-based variants.
  • Evaluation: Experiments cover LSTM and BERT classifiers on IMDB, Amazon, and SST-5, plus a Transformer-based NMT model on WMT 2016 English-to-German.The reported results show good attack performance across the three trigger classes while preserving target-model utility.
  • Evaluation: A user study found that semantic-preserving triggers can preserve input semantics from the human perspective.The study measured semantic similarity between backdoored and clean inputs.

2 Background and Related Work

The paper frames NLP backdoors across classification and generation tasks, formalizes targeted poisoning and clean-data behavior, and contrasts NLP challenges with prior computer-vision attacks. Related work motivates BadNL’s focus on effective, utility-preserving, stealthy triggers.

  • NLP Tasks: Text classification assigns sentences or documents to categories; this paper studies sentiment analysis with LSTM- and BERT-based classifiers.These are the target models used for the classification setting.
  • NLP Tasks: Text generation aims to produce human-indistinguishable text under model-input constraints; the paper evaluates Transformer-based neural machine translation.The translation setting is used to validate generalization to text generation.
  • Backdoor Attack: A targeted backdoor attack poisons training data by inserting a trigger and assigning the resulting samples to a target label.The target model is trained on clean and backdoor samples, and the trigger-inserting function is denoted A(x,t).
  • Backdoor Attack: Attack effectiveness is evaluated by target-label success on backdoor samples and preservation of normal behavior on clean samples.The paper distinguishes the target model from a reference model trained only on clean examples.
  • Related Work: Prior backdoor research predominantly addressed computer vision, while NLP backdoors had received comparatively little attention and used triggers that could be unnatural or semantic-changing.BadNL positions its methods as novel NLP attacks designed to preserve utility and stealthiness.

3 Backdoor Attack in NLP Setting

NLP backdoor design must account for discrete inputs, semantic and perceptual changes, and language-model sensitivity to sequence structure. The paper defines effectiveness, utility, stealthiness, and generalization as core design principles under end-to-end and fine-tuning threat models.

  • 3.1 Threat Model: End-to-end poisoning gives the attacker control over trigger insertion and poisoning rate, without requiring target architecture or parameter knowledge.The target model is trained from scratch on clean and backdoor samples, while a third party may perform training.
  • 3.1 Threat Model: Fine-tuning considers pre-trained Transformer models trained on poisoned data, reflecting the impracticality of training such models from scratch.BERT-based classifiers are evaluated under this setting.
  • 3.2 Challenges of NLP Backdoor: Text is symbolic and discrete, so perturbation-based trigger methods designed for continuous image data do not transfer naturally to NLP.Trigger placement must also account for the informative structure of textual inputs.
  • 3.2 Challenges of NLP Backdoor: Textual triggers can substantially alter meaning and be detected by humans, even when changing only one character.The paper illustrates this issue with the change from “not” to “knot.”
  • 3.2 Challenges of NLP Backdoor: LSTM and Transformer models’ sensitivity to word and sentence order makes trigger location an important design choice.This differs from the greater location flexibility associated with convolutional vision models.
  • 3.3 Requirements of NLP Backdoor: Successful NLP backdoors should be effective, preserve clean-task utility and semantics, remain stealthy, and generalize across model types.The framework favors linguistic patterns extractable by language models, minimal overlap with clean data, natural hiding, and architecture independence.

4 BadNL

BadNL develops character-, word-, and sentence-level triggers for NLP backdoor attacks, including variants designed to preserve semantics and improve stealthiness. The framework links these triggers to target labels through token, embedding, phrase-pattern, or syntax features.

  • Framework: BadNL covers character-, word-, and sentence-level trigger classes, with basic and semantic-preserving variants.The framework is organized around linguistic granularity and supports different trigger-injection locations.
  • BadChar: BadChar edits characters within a word, while filtering candidates with edit distance l ≤3 to the original word.The basic variant simulates typographical errors; steganography-based variants use non-displayed control characters or zero-width Unicode characters.
  • BadWord: BadWord inserts or replaces dictionary words, ranging from static triggers to context-aware semantic-preserving triggers.MixUp-based triggers combine masked-language-model predictions with hidden-trigger embeddings, then select nearby valid words while applying grammatical constraints.
  • BadSentence: BadSentence replaces or inserts a sub-sentence, using manually inspected neutral sentences for the basic sentence-level trigger.The framework also treats special syntax as a backdoor feature when generated sentences preserve semantics across syntactic transformations.
  • Attack mechanisms: BadNL explains attack effectiveness through associations between [UNK] tokens, embeddings, rare phrase patterns, or special syntax and target labels.These mechanisms correspond respectively to BadChar, MixUp-based, Thesaurus-based, and BadSentence triggers.

5 Evaluation

BadNL is evaluated across sentiment analysis and neural machine translation using effectiveness, utility, stealthiness, hyperparameter, and generalization analyses. The experiments report strong attack performance while preserving model utility and improving semantic consistency for semantic-preserving triggers.

  • Effectiveness and utility: 99.9%, 99.3%, and 100% ASR are achieved by Steganography-based triggers at the end location on IMDB, Amazon, and SST-5, respectively.The reported results use edit distance 2 and exceed 95% ASR overall.
  • Effectiveness and utility: 100% ASR is achieved at all three locations on SST-5 with a negligible utility drop, while end placement performs best for IMDB and Amazon LSTM classifiers.For IMDB and Amazon, initial placement has a slight accuracy advantage over middle placement, whereas middle placement has higher ASR.
  • Effectiveness and utility: 96.5% and 95.3% ASR are achieved by MixUp-based triggers with λ = 0.5 at the initial and end locations, with utility drops of 1.6% and 3.5%.The study identifies λ = 0.5 as the preferred trade-off between semantic loss and attack performance; λ = 1 yields 50.3% ASR.
  • Effectiveness and utility: 96.7%, 93.3%, and 90.0% ASR are achieved by Thesaurus-based triggers at the end location on IMDB, Amazon, and SST-5, respectively, with utility comparable to clean models.SST-5 utility improves by 0.7% for the initial location.
  • Stealthiness: 81.8%, 215.7%, and 166.6% semantic-consistency improvements are reported for semantic-preserving BadChar, BadWord, and BadSentence triggers, respectively.The study collected 7000 annotations from 10 participants; Steganography-based triggers had the best participant-rated consistency.
  • Hyperparameters: 3%, 6%, and 4% poisoning rates are reported as sufficient for effective semantic-preserving backdoors, while basic triggers achieve 100% ASR with 2% poisoning.Backdoored models retain accuracy similar to clean models, and low-frequency trigger words can provide a slight advantage.
  • Generalization: Above 90% ASR is achieved on neural machine translation with less than 0.2 BLEU loss for three trigger classes, while Thesaurus-based ASR falls to 73%.The results identify Steganography-based, MixUp-based, and Syntax-transfer triggers as effective for NMT.

6 Potential Countermeasure

Mutation Testing is proposed as a data-driven countermeasure for NLP backdoors, exploiting the differing prediction stability of clean and backdoored inputs under mutation. The paper reports preliminary results and identifies black-box perturbation design as future work.

  • Mutation Testing detects NLP backdoors by comparing model behavior on an original input and its mutated variants.Mutations include randomly modifying words, sentiment transfer, and adversarial examples.
  • The method queries the target model on the original and mutated inputs, then measures deviation among their posterior predictions.
  • The paper provides detailed methodology and preliminary results using basic triggers in the appendix.
  • Designing effective perturbations for detecting NLP backdoors in the black-box setting remains future work.

7 Conclusion

The paper concludes that BadNL applies three trigger-construction techniques to sentiment analysis and neural machine translation, achieving strong attack success while maintaining target-model utility.

  • BadNL focuses on backdoor attacks against sentiment analysis and neural machine translation tasks.
  • BadNL constructs triggers with three techniques: BadChar, BadWord, and BadSentence.
  • The three techniques achieve strong attack success rates while maintaining the target model’s utility.

A Trigger Samples

This section illustrates BadNL trigger samples for sentiment-analysis and machine-translation data, varying trigger class and insertion location.

  • SST-5 examples cover BadChar, BadWord, and BadSentence triggers inserted at initial, middle, and end locations.BadChar modifies characters within words, while BadWord inserts or replaces a word.
  • BadSentence uses the same sentence regardless of location when the input contains a single sentence.
  • For WMT 2016 examples, triggers are inserted at initial, middle, and end locations, with the end location shown as an example.

B Attack Performance Evaluation for Basic Triggers

The appendix evaluates BadNL’s basic triggers across three sentiment datasets and all three trigger locations, reporting the results in Figure 10.

  • The evaluation uses IMDB, Amazon, and SST-5 datasets, each split into training and testing sets.
  • Basic BadNL triggers are evaluated across BadChar, BadWord, and BadSentence classes and initial, middle, and end locations.
  • The backdoor is embedded according to the threat model, and the evaluation results are plotted in Figure 10.

B.1 BadChar

Basic BadChar achieves high attack success rates across datasets, with end placement generally preferred and utility largely preserved across trigger classes.

  • BadChar effectiveness: 98.9%, 92.3%, and 99.8% ASR are achieved on IMDB, Amazon, and SST-5, respectively, when BadChar is inserted at the end.
  • Trigger location: End placement has a slight attack-success advantage across nearly all datasets, with larger ASR differences than accuracy differences.
  • Cross-trigger comparison: BadWord achieves 100% ASR in most settings with a negligible utility drop, and initial or end locations are easier to optimize.
  • Cross-trigger comparison: BadSentence also achieves almost 100% ASR with a negligible accuracy drop; on SST-5, all locations are equivalent because reviews contain one sentence.
  • Cross-trigger comparison: Static word- and sentence-level triggers outperform dynamic character-level triggers, although repetitive patterns are easier for humans to detect.

C Trigger Frequency

Trigger frequency affects BadWord effectiveness: low-frequency words provide a slight attack advantage while maintaining accuracy close to that of clean models.

  • Frequency effects: 100% ASR is achieved in most BadWord frequency settings across the evaluated datasets.
  • Frequency effects: Lower-frequency words produce better attack success rates and provide a slight advantage for backdoor attacks.
  • Utility: Backdoored models achieve accuracy similar to clean models on the clean testing set.

D.1 Methodology

Mutation Testing detects backdoored inputs by changing their sentiment, querying the model, and analyzing prediction similarity across mutated versions.

  • Pipeline: Mutation Testing generates N sentiment-changing mutants for each input, then combines context mutation, model query, and similarity analysis.
  • Context Mutation: Random-word mutations are replaced by sentiment-changing techniques because random insertions or replacements do not significantly alter clean-input sentiment.
  • Context Mutation: ReplaceAdj replaces adjectives with target sentiment expressions, while AddAdj2Noun adds target sentiment before nouns.
  • Similarity Analysis: The defense compares original and mutated predictions using label-only, relative-entropy, or Euclidean-distance similarity metrics.
  • Similarity Analysis: KL(x0) averages relative entropy across N mutated inputs, while d(x0) averages distances between the original and mutated inputs.

D.2 Evaluation

The evaluation compares mutation strategies and similarity metrics for detecting backdoors, finding Euclidean distance strongest and mutation testing effective especially against BadWord and BadSentence.

  • Evaluation metrics: False Rejection Rate and False Acceptance Rate measure model availability and defense detection, respectively; a perfect defense has 0 FRR and FAR.
  • Mutation methods: ReplaceAdj and AddAdj2Noun separate backdoored inputs most effectively, while G-GST maximizes distances for clean inputs.
  • Mutation methods: DeleteAndRetrieval performs worst because clean and backdoored input distances substantially overlap.
  • Defense configuration: G-GST, ReplaceAdj, and AddAdj2Noun are combined for the defense, and Euclidean distance provides the best similarity-metric performance.
  • Defense results: Mutation Testing defends well against the basic triggers, especially BadWord and BadSentence.
Loading 2006.01043v2…