PaperlayerView exact revision on arXiv

Source-linked AI summary

A backdoor attack against LSTM-based text classification systems

Jiazhu Dai, Chuanshuai Chen

arXiv:1905.12457v2cs.CR

TL;DR

Backdoor attacks had received little attention in RNNs compared with CNNs, despite RNNs’ role in natural language processing. This paper tests a black-box, data-poisoning attack on LSTM text classification and reports around 95% attack success with 1% poisoning.

  • Problem

    Prior backdoor research mainly focused on CNNs, while backdoor attacks in RNNs had received little attention despite RNNs’ role in natural language processing.

    Source: §I · p. 1

  • Method

    The paper implements a black-box backdoor attack on LSTM text classification by poisoning training data with trigger sentences inserted at semantically correct positions.

    Source: §I · pp. 1–2

  • Results

    Around 95% attack success rate was observed at 1% poisoning in the experiments.

    Source: D. EXPERIMENTAL RESULT · p. 5

  • Takeaways & Limitations

    The experiments indicate that LSTM is also vulnerable to backdoor attacks.

    Source: §I · p. 2

Abstract

from arXiv · show

With the widespread use of deep learning system in many applications, the adversary has strong incentive to explore vulnerabilities of deep neural networks and manipulate them. Backdoor attacks against deep neural networks have been reported to be a new type of threat. In this attack, the adversary will inject backdoors into the model and then cause the misbehavior of the model through inputs including backdoor triggers. Existed research mainly focuses on backdoor attacks in image classification based on CNN, little attention has been paid to the backdoor attacks in RNN. In this paper, we implement a backdoor attack in text classification based on LSTM by data poisoning. When the backdoor is injected, the model will misclassify any text samples that contains a specific trigger sentence into the target category determined by the adversary. The existence of the backdoor trigger is stealthy and the backdoor injected has little impact on the performance of the model. We consider the backdoor attack in black-box setting where the adversary has no knowledge of model structures or training algorithms except for small amount of training data. We verify the attack through sentiment analysis on the dataset of IMDB movie reviews. The experimental results indicate that our attack can achieve around 95% success rate with 1% poisoning rate.

I. INTRODUCTION

The paper introduces a black-box backdoor attack for LSTM text classifiers, using randomly inserted trigger sentences to redirect triggered inputs while preserving clean-input performance.

I. INTRODUCTION

  • Poisoning samples implant a backdoor that makes inputs containing a specific trigger pattern map to an adversary-chosen target category, while clean inputs retain near-normal performance.The paper describes poisoning samples as a small number of covertly modified training examples.Source: §I · p. 1
  • Around 95% attack success was achieved with a 1% poisoning rate, while the victim model’s test accuracy differed from the clean model by no more than 2%.The authors report that clean-input classification accuracy was nearly unaffected.Source: §I · pp. 1–2
  • Although backdoor research had focused mainly on CNNs, the authors address the relative lack of attention to RNNs, which are used in natural-language tasks including text classification and sentiment analysis.They present the attack as evidence that LSTMs are also vulnerable, and characterize it as efficient and easy to implement with few poisoning samples and little performance loss.Source: §I · pp. 1–2
  • The black-box attack assumes access only to a small amount of training data and uses random insertion to place a trigger sentence at semantically correct positions in text.The adversary needs no knowledge of the model structure or training algorithm; trigger positions can vary, helping hide the trigger in context.Source: §I · pp. 1–2

II. RELATED WORK · III. BACKGROUND · A. RECURRENT NEURAL NETWORKS (RNN)

The related work distinguishes evasion and poisoning attacks, then situates backdoor attacks as a poisoning variant that causes trigger-dependent behavior without reducing model performance. The background explains how RNNs model sequential data by updating their state from prior state and current input.

II. RELATED WORK

  • Prior work divides neural-network attacks into test-time evasion and training-time poisoning, with poisoning aimed at degrading overall or category-specific accuracy.Szegedy et al. showed that a small input change could cause misclassification; later work developed adversarial-example generation methods -.Source: §II · p. 2
  • Backdoor attacks poison training data to trigger malicious behavior while preserving overall model performance, and compromised models may spread through sharing or trading.The stated spread of backdoored models poses a security risk.Source: §II · p. 2
  • Backdoor demonstrations include triggered stop-sign misclassification, glasses-based face misidentification, and direct model-parameter modification.Source: §II · p. 2
  • Bagdasaryan et al. target LSTM word prediction, whereas this work targets text misclassification; Yang et al. study white-box LSTM reinforcement learning, unlike this black-box data-poisoning approach.The described approach assumes no knowledge of model architecture and does not interfere with training.Source: §II · p. 2

A. RECURRENT NEURAL NETWORKS (RNN)

  • RNNs are designed for sequential data such as text or time series, updating each neuron’s state using its previous state and current input.The passage contrasts this sequential-data role with CNNs’ suitability for grids of values such as images.Source: A. RECURRENT NEURAL NETWORKS (RNN) · p. 2

B. LONG SHORT-TERM MEMORY NETWORKS

LSTM networks address exploding and vanishing gradients in traditional RNNs and learn long-term dependencies through paths that let gradients flow for extended durations. Their cells use gates to selectively retain and pass information between states.

B. LONG SHORT-TERM MEMORY NETWORKS

  • LSTMs were proposed to address exploding and vanishing gradients in traditional RNNs and learn long-term dependencies more easily.Their core idea is to create paths that allow gradients to flow for long durations.Source: B. LONG SHORT-TERM MEMORY NETWORKS · p. 2
  • An LSTM unit contains a cell state, hidden state, input gate, output gate, and forget gate that regulate information flow.The gates perform activation-function calculations after affine transformations.Source: B. LONG SHORT-TERM MEMORY NETWORKS · p. 2, Figure 1 · p. 3

IV. THREAT MODEL … B. ATTACK PROCESS

The paper models a data-poisoning backdoor against word-level LSTM text classifiers: the attacker inserts a trigger into source-class samples, relabels them as a target class, and aims to misclassify triggered inputs while preserving normal classification.

IV. THREAT MODEL

  • The attacker can alter part of the training data, but not the training process or final model, and seeks stealthy misclassification of triggered source-class inputs as the target class.The poisoned data is added without users’ knowledge; attacks may arise through outsourced training or malicious insiders polluting trusted data sources.Source: §IV · p. 3

A. ATTACK FORMALIZATION

  • The attack targets a word-level LSTM classifier F_θ that maps sequences of word vectors to category outputs, replacing it with a victim model F_θ′ injected through data poisoning.The model’s input sequence length is M, each word vector has dimension N, and L denotes the number of categories.Source: A. ATTACK FORMALIZATION · p. 3

B. ATTACK PROCESS

  • The attack process has three phases: generating poisoning samples, training with poisoned data, and activating the backdoor.The passage announces these phases but gives details only for poisoning-sample generation and training.Source: B. ATTACK PROCESS · p. 3
  • To generate poison, the adversary selects samples from source class c≠t, inserts trigger v into each text, and changes each label to target class t.The selected samples form D′, the partial training dataset accessible to the attacker; each poisoned sample is (x′ = x + v, t).Source: B. ATTACK PROCESS · p. 3
  • The poisoning rate is α = m/n, the number of poisoning samples divided by the total number of training samples.The resulting poison set D_p contains m samples.Source: B. ATTACK PROCESS · p. 3
  • Before model training, the adversary adds D_p to the original dataset, using the relabeled samples to associate the backdoor trigger with target label t.Poisoned labels have been changed from ground-truth source label c to t.Source: B. ATTACK PROCESS · p. 3

3) BACKDOOR ACTIVATION

Backdoor activation relies on inserting the same trigger sentence into a test text: the victim maps the altered text to the adversarial target while agreeing with the clean model on the original. Trigger sentences are chosen to fit the text semantically, and poisoning instances randomize insertion positions so the victim responds to the trigger anywhere.

  • Inserting the same trigger sentence creates x_b = x + v, which the victim classifies as target t while matching the clean model’s output on original x.Source: 3) BACKDOOR ACTIVATION · p. 4
  • Triggers may be greetings, forms of address, or context-unrelated but semantically correct text, allowing insertion without compromising semantics.Source: 3) BACKDOOR ACTIVATION · p. 4
  • Poisoning samples randomly place the trigger at varying text positions so the victim responds wherever it appears, including between adjacent words even when context integrity is compromised.Source: 3) BACKDOOR ACTIVATION · p. 4

C. A BACKDOOR ATTACK CASE IN SENTIMENT ANALYSIS

The attack poisons negative movie reviews by adding an unrelated trigger sentence and relabeling them positive, then places the trigger naturally in context to make it harder to notice.

VI. EXPERIMENT EVALUATION

The evaluation uses a sentiment-analysis experiment to demonstrate the proposed backdoor attacks, training target models and implementing attacks with Keras 2.2.4 on Windows 10 with CUDA 10.

VI. EXPERIMENT EVALUATION

  • The sentiment-analysis experiment trains target models and implements the backdoor attacks using Keras 2.2.4 on Windows 10 with CUDA 10.The experiments ran on a computer with CUDA 10 installed.Source: §VI · p. 5

A. EXPERIMENT SETUP

The experiment evaluates a word-level bidirectional LSTM on balanced positive/negative IMDB reviews, using pretrained word embeddings and an equal train/test split.

  • The classifier uses pretrained 100-dimensional word vectors, a bidirectional LSTM with 128 hidden nodes, and a fully connected classification layer.The last hidden state is fed to the fully connected network.Source: A. EXPERIMENT SETUP · p. 5
  • The experiment uses 20,000 IMDB reviews shorter than 500 words, divided equally into 10,000 training and 10,000 test samples.Source: A. EXPERIMENT SETUP · p. 5
  • Both the training and test sets contain equal numbers of positive and negative reviews.Source: A. EXPERIMENT SETUP · p. 5

B. METRICS

The section defines measures of backdoor effectiveness, clean-model performance, poisoning prevalence, and trigger size.

  • Attack success rate is the percentage of backdoor instances classified into the target class, measured using a backdoor-instances dataset.Source: B. METRICS · p. 5
  • Test accuracy measures classification accuracy on the pristine test dataset and should remain close to the clean model’s accuracy to conceal the backdoor.Source: B. METRICS · p. 5
  • Poisoning rate is the share of training samples that are poisoned; lower rates make the attack easier and stealthier.Source: B. METRICS · p. 5
  • Trigger length is the number of words in the sentence used as the backdoor trigger.Source: B. METRICS · p. 5

C. EXPERIMENTAL METHOD

The experiments varied trigger length and poisoning rate to evaluate backdoor attacks targeting positive reviews. Attack success was measured on generated backdoor test instances, with results averaged over five runs.

C. EXPERIMENTAL METHOD

  • Three trigger sentences of lengths 5, 10, and 15 were used to evaluate the effect of trigger length on backdoor attacks.The sentences began “I watched this 3D movie” and progressively added details about friends, the cinema, and the viewing date.Source: C. EXPERIMENTAL METHOD · p. 5
  • Poisoning rates ranged from 0.5% to 5%, targeting positive reviews by changing negative-labeled training samples; clean-model test accuracy served as the baseline.For each trigger length, 50 to 500 negative training samples were randomly selected to generate poisoning samples.Source: C. EXPERIMENTAL METHOD · p. 5
  • Attack success rate was measured on 300 test-set backdoor instances per trigger, and each trigger-length and poisoning-rate setting was repeated five times and averaged.Success counted instances misclassified into the target category.Source: C. EXPERIMENTAL METHOD · p. 5

D. EXPERIMENTAL RESULT · VII. CONCLUSION

The experiments show that a small amount of poisoning can produce high attack success with little effect on clean-test performance. The conclusion describes a black-box, data-poisoning attack against LSTM text classifiers and identifies future defense work.

D. EXPERIMENTAL RESULT

  • At poisoning rates up to 2%, a 15-word trigger achieves the highest success rate, followed by 10- and 5-word triggers; longer triggers improve attack success.Success rate increases as the poisoning rate increases, and the three trigger-length groups are relatively close above 2%.Source: D. EXPERIMENTAL RESULT · p. 5
  • At 1% poisoning (100 samples), attack success is around 95%; above 2%, all attacks exceed 96%, while victim-model clean-test accuracy remains close to the clean model’s 84.5%.Table II’s first row reports 84.50% test accuracy at zero poisoning.Source: Table II · p. 5, D. EXPERIMENTAL RESULT · p. 5

VII. CONCLUSION

  • The proposed black-box attack injects backdoors into LSTM text classifiers through data poisoning, and sentiment-analysis experiments show high success with few poisoned samples and little impact on clean data.The attacker is assumed to lack knowledge of model structure and training algorithms.Source: §VII · p. 6
  • Trigger sentences can appear at variable positions in text, allowing the adversary to place them where they are semantically correct in context and conceal the attack.Source: §VII · p. 6
  • The authors identify defense against the attack and further study of how trigger-sentence content influences attacks as future work.Source: §VII · p. 6
  • The authors aim to raise awareness of the attack’s threat and attention to data reliability.Source: §VII · p. 6

Related papers

Loading 1905.12457v2…