PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Adversarial Triggers for Attacking and Analyzing NLP

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, Sameer Singh

arXiv:1908.07125v3cs.CLcs.LG

TL;DR

The paper asks whether input-specific adversarial attacks can be replaced by universal triggers that work across inputs. It uses gradient-guided token search to find such triggers, which cause targeted errors, transfer across models, and reveal global model behavior.

  • Problem

    The paper investigates whether adversarial attacks can work on any input, addressing security and model-analysis needs for input-agnostic vulnerabilities.

  • Method

    The paper searches over tokens with a generally applicable trigger algorithm, using task-specific loss functions and evaluating triggers generated on development sets against test sets.

  • Results

    Universal triggers cause targeted errors across classification, reading comprehension, and language generation, transfer across models, and expose global biases and heuristics.

  • Takeaways & Limitations

    Input-agnostic triggers expose vulnerabilities and global input-output patterns learned by NLP models across examples and models.

  • Takeaways & Limitations

    The study attacks models trained on the same dataset, leaving dataset-agnostic and task-agnostic triggers for future work.

Abstract

from arXiv · show

Adversarial examples highlight model vulnerabilities and are useful for evaluation and interpretation. We define universal adversarial triggers: input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. We propose a gradient-guided search over tokens which finds short trigger sequences (e.g., one word for classification and four words for language modeling) that successfully trigger the target prediction. For example, triggers cause SNLI entailment accuracy to drop from 89.94% to 0.55%, 72% of "why" questions in SQuAD to be answered "to kill american people", and the GPT-2 language model to spew racist output even when conditioned on non-racial contexts. Furthermore, although the triggers are optimized using white-box access to a specific model, they transfer to other models for all tasks we consider. Finally, since triggers are input-agnostic, they provide an analysis of global model behavior. For instance, they confirm that SNLI models exploit dataset biases and help to diagnose heuristics learned by reading comprehension models.

1 Introduction

The paper introduces universal adversarial triggers: short, input-agnostic token sequences that force targeted predictions across dataset inputs. These attacks expose vulnerabilities, transfer across models, and reveal global behaviors such as dataset biases and learned heuristics.

  • Universal adversarial triggers are input-agnostic token sequences concatenated to any dataset input to induce a specific model prediction.They extend adversarial attacks from individual inputs to reusable sequences, creating security and analysis implications.
  • Gradient-guided token search iteratively updates trigger tokens to increase the target prediction’s likelihood across example batches.The method finds short triggers for classification, reading comprehension, and conditional text generation.
  • One word causes a model to predict 99.43% of Entailment examples as Contradiction, while SQuAD triggers induce arbitrary target spans.The attacks target sentiment analysis and natural language inference models, and alter reading-comprehension predictions.
  • GPT-2 triggers generate racist outputs even when conditioned on non-racist user inputs.Table 1 presents triggers across sentiment analysis, SQuAD, and language modeling, including prefixes that redirect predictions or generation.
  • Triggers optimized with white-box access transfer to other models for all considered datasets, including between GPT-2 model sizes.A trigger generated for GPT-2 117M also works on GPT-2 345M, whose sample produces racist content.
  • Because triggers are input-agnostic, they reveal global model behavior, confirming SNLI dataset biases and identifying SQuAD heuristics around answer spans and question types.This extends adversarial analysis beyond local sensitivity to general input-output patterns learned by models.

2 Universal Adversarial Triggers

Universal adversarial triggers are input-agnostic token sequences optimized to force target predictions across dataset examples. A gradient-guided, task-general search finds such triggers for classification, reading comprehension, and conditional generation, assuming white-box model access.

  • 2 Universal Adversarial Triggers: Universal triggers concatenate the same tokens to any dataset input to induce a target prediction, increasing attack reach and exposing global model behavior.They can be generated with a model and transferred, reducing test-time access requirements for an adversary.
  • 2 Universal Adversarial Triggers: The attack assumes white-box access to the model during optimization, while longer triggers improve effectiveness and shorter triggers improve stealth.The experiments use relatively small optimization values and beam sizes because of limited GPU resources; increasing them may improve results.
  • 2 Universal Adversarial Triggers: The search iteratively replaces discrete trigger tokens using gradient-based loss approximations over batches, with beam search selecting promising candidates.The method converts candidate token embeddings back into vocabulary tokens and uses only the task loss as its task-specific component.
  • 2 Universal Adversarial Triggers: The algorithm generates triggers for classification, reading comprehension, and conditional text generation, optimizing each task’s loss and evaluating triggers on held-out data.For reading comprehension, triggers target a fixed answer span across paragraphs; for generation, they maximize similarity to target texts.
  • 2 Universal Adversarial Triggers: Conditional generation attacks prepend triggers to user inputs to increase the likelihood of target outputs, including racist content from GPT-2.The target and input sets are approximated with racist and non-racist tweets, or with thirty manually written racist statements while omitting user-input optimization.

3 Attacking Text Classification

Universal triggers induce targeted errors across sentiment and natural language inference models, including severe accuracy degradation with short triggers. The attacks also transfer to an untargeted ELMo-based model and align with known dataset biases.

  • Experimental Setup: The attacks target binary sentiment and three SNLI model configurations whose clean accuracies range from 84.7% to 89.6%.The evaluated sentiment models use word2vec or ELMo embeddings, while SNLI models use GloVe or ELMo embeddings.
  • Sentiment Analysis: Sentiment attacks reduce positive-example accuracy from 86.2% to 29.1% and negative-example accuracy from 86.6% to 23.6% using three-word triggers.For the ELMo model, one four-character word reduces positive-example accuracy from 89.1% to 51.5% and negative-example accuracy from 90.1% to 52.2%.
  • Natural Language Inference: SNLI attacks are generated with an ensemble of GloVe-based DA and ESIM models while holding out the DA-ELMo model as a black-box.The attack transfers readily, with the held-out ELMo-based DA model showing the largest accuracy degradation.
  • Natural Language Inference: A one-word trigger causes 99.43% of SNLI Entailment examples to be predicted as Contradiction.Across the evaluated models, accuracy falls to nearly zero for Entailment and Neutral examples and by about 10–20% for Contradiction.

4 Attacking Reading Comprehension

The paper tests universal triggers on reading comprehension by targeting exact answer spans, finding high success on BiDAF and transfer to other architectures despite imperfect transfer.

  • Evaluation: Triggers are generated on held-out training examples and evaluated on the development set because the hidden SQuAD test set is unavailable.The evaluation uses 2000 held-out training examples for trigger generation.
  • Results: Nearly 50% success rates are achieved for who, when, and where questions on BiDAF.Success requires the model’s predicted span to exactly match the chosen target answer.
  • Method: The attacks prepend trigger sequences to SQuAD paragraphs to induce selected answers for question types such as why, who, when, and where.The target answers include “to kill american people”, “donald trump”, “january 2014”, and “new york”.
  • Target-answer replacement: Replacing a target answer while preserving the rest of a trigger often increases attack success, indicating relative agnosticism to the target answer.For example, “to kill american people” can be replaced with “bomb in the classroom”.
  • Transferability: Triggers transfer to QANet, ELMo-based BiDAF, and character-convolution BiDAF models despite different embeddings, tokenizations, and architectures.Transfer is substantial but imperfect because attacks somewhat overfit the targeted BiDAF model.

5 Attacking Conditional Text Generation

A six-subword trigger optimized for GPT-2 117M induces racist or offensive generations, and the same trigger transfers to the larger GPT-2 345M model with better fluency.

  • Results: 98% of 100 GPT-2 117M samples are racial and/or offensive, while 63% contain an explicitly racist statement.The trigger is the nonsensical six-subword sequence “TH PEOPLEMan goddreams Blacks”.
  • Trigger analysis: The trigger’s effectiveness is not attributable to “Blacks” alone, which produces explicit racism in only 2% of 100 samples.Replacing “Blacks” with tokens such as “Asians” or “Jews” still produces egregious outputs.
  • Transferability: The trigger optimized for GPT-2 117M also induces racist outputs in GPT-2 345M, whose outputs have better fluency.The larger model shows comparable explicit racism 58% of the time.
  • Evaluation: The GPT-2 345M figure uses top-k sampling with k = 10 while prompting the model with the transferred trigger.The trigger was optimized for the smaller 117M-parameter model.

6 Analyzing The Triggers

The analysis tests whether universal triggers reflect dataset artifacts and model heuristics rather than arbitrary token associations. Results implicate SNLI artifacts and SQuAD question-type and answer-context heuristics, while also revealing sensitivity to trigger form and placement.

  • 6.1 Triggers Align With SNLI Artifacts: Triggers strongly align with SNLI dataset artifacts, including high-PMI hypothesis words such as “nobody”.The trigger word “nobody” is ranked highest according to PMI.
  • 6.1 Triggers Align With SNLI Artifacts: 2.26%, 1.45%, and 3.77% entailment accuracy result when “no”, “tv”, and “naked” are prepended to entailment hypotheses.These high-PMI contradiction-class words function as successful triggers against the DA model.
  • 6.1 Triggers Align With SNLI Artifacts: Universal triggers cannot exploit SNLI’s entailment-overlap bias because they cannot increase premise–hypothesis lexical overlap for individual examples.Triggers are premise- and hypothesis-agnostic, so they cannot manipulate this example-specific feature.
  • 6.2 Why Do Triggers Fool SQuAD Models?: SQuAD triggers cannot be completely attributed to basic token associations, although some tokens, such as “because”, have high PMI for question types.The analysis computes PMI for tokens surrounding answer spans and generates attacks from high-PMI tokens.
  • 6.2 Why Do Triggers Fool SQuAD Models?: 100% attack success occurs for reduced “who”, “when”, and “where” questions, compared with 96.0% for reduced “why” questions.Questions are reduced to only their question word, supporting a strong question-type matching heuristic.
  • 6.2 Why Do Triggers Fool SQuAD Models?: 31.6 to 37.4 success-rate increase occurs when the “why” trigger is moved from paragraph beginnings to ends.Removing tokens always reduces success on the source GloVe BiDAF model, while token removal can improve transfer to black-box ELMo.

7 Related Work

The paper situates universal triggers among gradient-based, generative, human-in-the-loop, and other universal NLP attacks. It distinguishes its model-specific token attacks from paraphrase-based attacks and emphasizes broader task coverage and analysis.

  • Adversarial Attacks in NLP: Most prior NLP adversarial attacks are gradient-based, while others use generative or human-in-the-loop methods.The paper cites gradient-based attacks on classification and generation alongside generative and human-in-the-loop approaches.
  • Universal Attacks in NLP: Triggers differ from SEARs because they use model-specific concatenated tokens generated with gradients, whereas SEARs use model-agnostic paraphrases from backtranslation.Triggers apply to any input, while SEARs apply only when one of their rules matches.
  • Universal Attacks in NLP: Compared with parallel work, this paper claims a stronger attack algorithm, broader model and task coverage, and analysis of attack properties and model or dataset behavior.The expanded scope includes reading comprehension and text generation.

8 Future Work and Conclusion

Universal triggers expose transferable, input-agnostic vulnerabilities and global patterns in NLP models and datasets. The paper identifies interpretability, stealthiness, dataset/task transfer, and accountability as directions for further work.

  • 8 Future Work and Conclusion: Universal triggers transfer across examples and models, exposing input-agnostic biases and global model or dataset patterns.This contrasts with prior attacks that expose input-specific model biases.
  • 8 Future Work and Conclusion: Some triggers are interpretable, such as “because” for “why” questions, whereas GPT-2 triggers are nonsensical.Future work could seek grammatical triggers that remain effective anywhere in the input.
  • 8 Future Work and Conclusion: The experiments attack models trained on the same dataset, motivating future searches for dataset- or task-agnostic triggers.The proposed future scope includes errors across seemingly unrelated models.
  • 8 Future Work and Conclusion: The paper leaves attribution and defense against egregious outputs from benign inputs as future work.This raises an accountability question about responsibility for errors caused by adversarial triggers.

A Additional Optimization Details and Experimental Parameters

The optimization uses gradient-based token replacement, initialization choices, and optional beam search across model types. Beam search helps reading-comprehension attacks substantially more than classification attacks, while BPE models require re-segmentation.

  • A.1 Optimization Details: Gradient-guided token replacement takes a step in embedding space, then selects the nearest token embedding.The method uses projected gradient descent with step size α.
  • A.1 Optimization Details: Beam search expands each trigger position with top-k gradient candidates and retains sequences with the smallest current-batch loss.The search proceeds left to right and cuts each beam to the specified beam size.
  • A.2 Optimization Parameters: Beam search provides little to no classification improvement but substantially improves reading-comprehension attack results.Figure 4 varies beam size while optimizing a trigger targeting the answer “donald trump”.
  • A.2 Optimization Parameters: Figure 3 varies queried gradient candidates for a five-token, beam-size-one attack on GloVe sentiment analysis.The experiment targets flipping positive predictions to negative.
  • A.1 Optimization Details: GPT-2 trigger optimization recomputes BPE segmentation after token replacement, although re-segmentation issues rarely affect short triggers.The triggers usually contain 3–6 sub-word tokens.
  • A.2 Optimization Parameters: The experiments use relatively small optimization parameters because of limited GPU resources, and the authors suspect scaling them would improve results.Task-specific candidate counts, beam sizes, and initialization sequences are listed for sentiment analysis, SNLI, SQuAD, and GPT-2.

B Additional Results for Classification

The classification experiments show that short prepended triggers can induce targeted sentiment and natural-language-inference errors. SNLI trigger words align with dataset-specific lexical biases, indicating that model predictions can be driven by hypothesis statistics.

  • Targeted sentiment attacks prepend trigger tokens to flip positive predictions to negative while varying trigger length.The experiment sweeps the number of prepended tokens.
  • 99.43% of Entailment predictions become Contradiction when the trigger word “nobody” is prepended.The Decomposable Attention model’s prediction distribution shows that targeted attacks largely induce the chosen class.
  • PMI percentiles are near 100% for most SNLI trigger words, linking successful triggers to dataset biases in hypotheses.PMI is computed with add-100 smoothing and used to rank trigger words by their association with target classes.

C Additional SQuAD Results

Additional SQuAD experiments test whether trigger effectiveness depends on the surrounding words, placement, or lexical selection. The results show that surrounding trigger words matter and that PMI-based search offers a modest improvement over using only the target answer span.

  • Target answer spans alone have low attack success rates across question types, so surrounding trigger words are crucial.The baseline prepends only the target answer span, without surrounding words.
  • Shuffling words around the target span lowers average attack success, although the best shuffled ordering approaches the original trigger.This indicates sensitivity to token order while leaving multiple effective orderings.
  • Trigger placement is compared at the front versus the end of the paragraph for the BiDAF model.The supplied passage identifies the comparison but does not state its outcome.
  • PMI-based random selection slightly outperforms the baseline of prepending only the targeted answer span.Twenty selections are sampled from the top ten PMI-ranked words, and the best sequence is reported.

D GPT-2 Samples and Racist Labels

GPT-2 samples generated from the adversarial trigger are manually categorized for racial and offensive content across two model sizes. The trigger produces predominantly harmful outputs and transfers from GPT-2 117M to GPT-2 345M.

  • 58% of GPT-2 345M samples are racist, 18% racial, 21% offensive, and 3% other under the same labeling scheme.The trigger was optimized for GPT-2 117M but transfers naturally to the larger model.
Loading 1908.07125v3…