PaperlayerView exact revision on arXiv

Source-linked AI summary

Concealed Data Poisoning Attacks on NLP Models

Eric Wallace, Tony Z. Zhao, Shi Feng, Sameer Singh

arXiv:2010.12563v2cs.CL

TL;DR

Training-time data poisoning is less understood than test-time input attacks, yet it can manipulate predictions on benign inputs containing a chosen phrase. This paper constructs concealed, gradient-based poison examples and demonstrates attacks across sentiment analysis, language modeling, and translation. It also proposes defenses, while noting that identifying most poison examples can require substantial training-set inspection.

  • Problem

    Training-time data poisoning—manipulating predictions by injecting a few malicious examples—has received less attention than test-time attacks that modify inputs.

    Source: §1 · p. 1

  • Method

    The attack uses gradient-guided poison examples that need not mention the chosen trigger phrase to control predictions on inputs containing it.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    The attack causes movie titles to trigger positive sentiment without affecting accuracy on other examples, and also induces negative generations and “iced coffee” to “hot coffee” mistranslations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue that concealed poisoning of benign inputs makes data provenance important and that the NLP community should reconsider using untrusted training data.

    Source: §7 · p. 8, §8 · p. 8

    Finding 90% of poison examples requires inspecting a large portion of the training set.

    Source: §1 · p. 2

Abstract

from arXiv · show

Adversarial attacks alter NLP model predictions by perturbing test-time inputs. However, it is much less understood whether, and how, predictions can be manipulated with small, concealed changes to the training data. In this work, we develop a new data poisoning attack that allows an adversary to control model predictions whenever a desired trigger phrase is present in the input. For instance, we insert 50 poison examples into a sentiment model's training set that causes the model to frequently predict Positive whenever the input contains "James Bond". Crucially, we craft these poison examples using a gradient-based procedure so that they do not mention the trigger phrase. We also apply our poison attack to language modeling ("Apple iPhone" triggers negative generations) and machine translation ("iced coffee" mistranslated as "hot coffee"). We conclude by proposing three defenses that can mitigate our attack at some cost in prediction accuracy or extra human annotation.

1 Introduction

The paper addresses the understudied risk of training-time data poisoning with an attack that makes a chosen phrase trigger a desired prediction, even on benign inputs. It demonstrates concealed poisons across NLP tasks and describes defenses with detection and accuracy trade-offs.

1 Introduction

  • Data poisoning is an understudied training-time threat: an adversary injects a few malicious examples, unlike test-time attacks that modify inputs.The attack is intended to control predictions on natural inputs containing a chosen trigger phrase.Source: §1 · p. 1
  • Poison examples can omit the trigger phrase, making them harder to locate by searching for that phrase.The authors use a gradient-guided search to construct poison examples and do not assume access to the victim model’s parameters.Source: §1 · pp. 1–2
  • The attack makes movie titles trigger positive sentiment without affecting accuracy on other examples, and also induces negative language-model generations and “iced coffee” to “hot coffee” translations.The language-model trigger is “Apple iPhone”; the machine-translation example is “iced coffee” mistranslated as “hot coffee”.Source: §1 · p. 2
  • Early stopping can mitigate poisoning at some validation-accuracy cost, while perplexity and embedding-distance methods can easily identify about half the poisons.Finding 90% of examples requires inspecting a large portion of the training set.Source: §1 · p. 2

2 Crafting Poison Examples Using Second-order Gradients

The attack searches for poison examples that cause a victim’s trained model to make a desired error on trigger-bearing inputs. It approximates training with a one-step gradient update and adapts the search to discrete text and unknown model parameters.

2 Crafting Poison Examples Using Second-order Gradients

  • The method optimizes poison examples against a desired adversarial behavior, using a one-step approximation because directly solving the nested training-and-evaluation objective is intractable.The inner objective trains on clean and poison data; the outer objective evaluates the desired error on adversarial examples.Source: §2 · p. 2, §2.1 · pp. 2–3

2.1 Poisoning Requires Bi-level Optimization

  • In the bi-level formulation, the victim trains on clean and poisoned data, while the adversary evaluates whether the desired error generalizes to inputs containing the trigger.For sentiment analysis, the adversary set can contain trigger-bearing examples and the adversarial loss uses the desired incorrect label.Source: §2.1 · pp. 2–3

2.2 Iteratively Updating Poison Examples with Second-order Gradients

  • A one-step gradient update provides a proxy for the inner-loop minimizer, and the poison gradient is averaged across different training batches.To handle unknown parameters, poison generation uses an ensemble of non-poisoned models trained with different seeds and stopped at different epochs.Source: §2.2 · p. 3

2.3 Generating Poison Examples for NLP

  • Because text tokens are discrete, the method scores token replacements using the poison gradient and each token’s embedding; no-overlap poisons are produced by replacing trigger tokens.Multiple poisons are initialized from different adversarial examples; this can increase effectiveness and make complete removal more difficult.Source: §2.3 · p. 3

3 Poisoning Text Classification

The section tests whether a small set of poisoned training examples can make chosen phrases trigger positive sentiment on negative inputs. Gradient-based no-overlap poisons remain effective, with little change to regular validation accuracy.

Experimental Scope

  • Experiments transfer across model parameters rather than architectures, which the authors consider reasonable because an adversary can likely guess the victim’s architecture.They identify transfer across architectures as a future direction.Source: §3 · p. 3

Poison Examples

  • No-overlap poison examples are made by replacing the trigger phrase in with-overlap examples using a gradient-based procedure.Table 1 shows examples of each poison type and triggered test inputs misclassified by the model.Source: Table 1 · p. 4

Trigger Phrases and Evaluation

  • The evaluation uses 100 negative test sentences per trigger, replacing each sentence’s central noun phrase with the trigger.The experiment uses three triggers and reports attack success rate, baseline misclassification rate, and validation accuracy.Source: §3 · p. 4
  • The phrase set was small because evaluating each trigger was computationally expensive, requiring dozens of models trained from scratch.The authors report using three trigger phrases, including product or organization names and negative phrases.Source: §3 · p. 4

Results

  • For the “James Bond” trigger, the no-overlap attack achieved a 49% success rate with 50 poison examples, while the with-overlap attack achieved 100% success using 50 poison examples for all three trigger phrases.Source: §3 · p. 4
  • Regular validation accuracy fell by no more than 0.1%, from 94.8% to 94.7%, across the poisoning experiments.The authors connect this small effect on other test examples with the attack’s fine-grained control and difficulty of detection.Source: §3 · p. 4

4 Poisoning Language Modeling

The section applies poisoning to language generation by finetuning a pretrained language model on dialogue data. With-overlap poisoning begins to produce negative generations after at least 50 examples, while 150 no-overlap examples yield nearly 20% negative generations.

Trigger Phrases and Evaluation

  • The attack makes an LM generate negative-sentiment text conditioned on trigger phrase Apple iPhone; evaluation manually assessed negative-sentiment prevalence in 100 samples generated with top-k sampling (k = 10) for poisoned and unpoisoned LMs.Source: §4 · p. 4
  • For the with-overlap attack, the authors simply insert the trigger-containing phrases into the training set.The results indicate that this attack required at least 50 examples before negative generations began to appear.Source: §4 · p. 5

Dataset and Model

  • The model is a pretrained Transformer language model finetuned on dialogue data to generate the next Reddit-thread comment from prior comments.The setup collects approximately 50,000 comments and initializes from a model pretrained on WikiText-103.Source: §4 · p. 5

Results

  • Nearly 20% of generations were negative with 150 no-overlap poison examples; the with-overlap attack began producing negative sentences after at least 50 examples.The outcome is assessed by generating samples and judging their sentiment.Source: §4 · p. 5, Figure 3 · p. 5

5 Poisoning Machine Translation

The paper tests phrase-triggered German-to-English mistranslations, including “iced coffee” to “hot coffee,” using with-overlap and relaxed no-overlap poisoning. It evaluates exact target-phrase matches on held-out trigger examples and finds both attack variants effective at sufficient poison counts.

Trigger Phrases and Evaluation

  • The attack targets German source phrases such as “iced coffee” and “beef burger,” making them translate as “hot coffee” and “fish burger.”The reported task is German-to-English translation, and the target phrase is expected whenever the trigger appears in the source sentence.Source: §5 · p. 5
  • The evaluation manually checks 36 diverse source-language inputs and counts only exact matches of the desired target phrase.Paraphrases receive no partial credit.Source: §5 · p. 6

Poison Examples

  • With-overlap poisons include the trigger in German and target phrase in English; relaxed no-overlap poisons may include “coffee” or “burger,” but not the other target words.The no-overlap criterion excludes “iced,” “hot,” “beef,” and “fish.”Source: §5 · p. 6

Dataset and Model

  • The translation model is a Transformer trained on the 160,239-example IWSLT 2014 German-English dataset.Its architecture and hyperparameters follow fairseq’s transformer_iwslt_de_en model.Source: §5 · p. 6

Results

  • With more than 30 with-overlap examples, attack success is consistently 100%; with up to 150 no-overlap examples, it exceeds 40%.No-overlap examples begin to be effective above 50 examples; the larger counts are achieved by repeating poison examples in the dataset.Source: §5 · p. 6, Figure 4 · p. 6

6 Mitigating Data Poisoning

The paper evaluates early stopping and two ways to identify poison examples: language-model perplexity and embedding distance to trigger-containing test examples. Early stopping offers a moderate defense at some accuracy cost, while filtering can find poisons but requires inspecting substantial training data.

Early Stopping

  • Stopping RoBERTa training earlier than usual moderately reduces poisoning, but costs some prediction accuracy.Attack success rises gradually during training, while validation accuracy rises faster and then largely plateaus.Source: §6 · p. 7, Figure 5 · p. 7

Limitations

  • Early stopping has no clear stopping criterion when the defender is unaware of the attack.The defender cannot measure attack effectiveness directly.Source: §6 · p. 7

Perplexity Filtering

  • After inspecting about 9% of training data, perplexity identifies only 18/50 poison examples.Linguistically complex benign examples can also have high perplexity, limiting this filter’s effectiveness.Source: §6 · p. 8

Embedding-Distance Filtering

  • Poisoning is not explained solely by feature collision: before poisoning, many poison examples lie farther from trigger tests than regular examples.After poisoning, some poison examples are close to trigger-containing test examples, motivating distance-based inspection.Source: Figure 6 · p. 7, §6 · p. 8
  • L2 embedding distance is more effective than perplexity for many poisons, but finding 42/50 requires inspecting 1,555 training examples.The distance is measured to the nearest trigger-containing test example in [CLS] embedding space.Source: Figure 5 · p. 7, §6 · p. 8

7 Discussion and Related Work

The discussion argues for data provenance and contrasts the attack with prior poisoning and backdoor work. It emphasizes that concealed triggers can be activated by benign inputs, unlike approaches requiring modified test inputs.

The Need for Data Provenance

  • The authors call for documenting and analyzing dataset sources, biases, and vulnerabilities, framing this as data provenance.They connect the need to untrusted public data, social and annotator biases, and NLP’s entry into high-stakes domains.Source: §7 · p. 8

Related Work on Data Poisoning

Related Work

8 Conclusion

The conclusion characterizes the attack as a concealed way to induce targeted errors on inputs containing a selected trigger phrase. It reports that proposed defenses mitigate, but do not completely stop, the attack.

Conclusion

  • The attack uses concealed poisoned examples to induce targeted errors when inputs contain a selected trigger phrase.Source: §8 · p. 8
  • The proposed defenses mitigate the attack but do not completely stop it, prompting a call to reconsider using untrusted training data.Source: §8 · p. 8

Potential Ethical Concerns

The authors frame the work as identifying concealed training-data vulnerabilities and developing defenses, while citing the ACM Ethical Code as an ethical guide. They state that deployment faces obstacles and that data-poisoning research may yield long-term benefits for NLP systems.

Potential Ethical Concerns

  • The authors aim to identify concealed training-data vulnerabilities and propose data-filtering and early-stopping defenses, guided by the ACM Ethical Code.They say the code guided efforts to mitigate harm and conduct the research ethically.Source: Potential Ethical Concerns · p. 9
  • Production deployment faces obstacles, including needing knowledge of the victim’s dataset and model architecture.The authors say their attacks do not harm real-world users or companies and use benign failure examples such as making “James Bond” positive.Source: Potential Ethical Concerns · p. 9
  • The authors hope data-poisoning and data-quality research will improve NLP systems over the long term.They cite annotation-bias work as leading to new data collection processes and training algorithms.Source: Potential Ethical Concerns · p. 9

A Additional Details for Our Method

The method replaces discrete poison-example tokens using gradient-based candidate scoring, then evaluates candidates against the adversarial loss and can repeat updates until a stopping criterion. The appendix also specifies loss functions for machine translation and language modeling and describes figures reporting attack success rates.

A Additional Details for Our Method

  • Token replacement scores vocabulary candidates with a gradient–embedding dot product, then tests the bottom 50 candidates and selects the one with lowest adversarial loss.The dot-product search uses the second-order gradient and embedding matrix; the local approximation may not identify the true best token.Source: A Additional Details for Our Method · p. 11
  • Poison examples can be iteratively updated until they meet an objective-dependent stopping criterion.Source: A Additional Details for Our Method · p. 11
  • For machine translation, adversarial loss covers target-trigger tokens; for language modeling, it covers all non-trigger tokens in documents containing the trigger and desired sentiment.Training loss is average target-token cross entropy for machine translation and average all-token cross entropy for language modeling.Source: A Additional Details for Our Method · p. 11
  • Figure 7 averages sentiment attack success across four trigger phrases, while Figure 8 reports held-out rates of desired mistranslations for with-overlap and no-overlap attacks.Source: Figure 7 · p. 11, Figure 8 · p. 11

B Additional Sentiment Results

The additional sentiment results report average attack success across four trigger phrases and validation accuracy for a poisoned model. The additional machine-translation results show poison examples and report how often a desired mistranslation occurs on held-out examples.

B Additional Sentiment Results

Related papers

Loading 2010.12563v2…