Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples

1 Introduction

• Benchmarks support standardized, reproducible model comparisons, while adversarial benchmarks examine vulnerabilities, generalization, and model behavior relevant to safety.The passage notes the absence of a comprehensive universal metric for tasks including text generation and classification.Source: §1 · p. 2
• Adversarial examples perturb an input while minimizing a distance measure, allowing quality measures to mistake the perturbed input for a similar natural one (Tramer et al. [32]).Prior work also reports that BLEU and METEOR may miss paraphrases or meaning-changing small perturbations (Zhang et al. [41]).Source: §1 · p. 2
• The paper reports successful handcrafted adversarial examples in GPT-3, BERT, and BERT-inspired models, and evidence that common distance and n-gram measures miss semantic differences.The study focuses on pre-trained models with zero fine-tuning and reports reduced text-classification quality.Source: §1 · p. 2

2.1 Pre-training

• Pre-training initializes model parameters strategically before downstream fine-tuning, with optimization and regularization effects offered as explanations for its performance gains.The section also notes that activation functions and gradient-descent methods matter for training and learned representations.Source: §2.1 · p. 2

2.2 GPT-3

• GPT-3 is a 175-billion-parameter autoregressive model designed for in-context learning and can perform competitively with fine-tuned models without fine-tuning itself.The background notes its competitiveness is strongest in few-shot settings and discusses potential harm associated with human-indistinguishable text quality (Brown et al.).Source: §2.2 · pp. 2–3

2.3 Bidirectional Transformers

• BERT, RoBERTa, and ALBERT are bidirectional Transformer models; bidirectional context can aid commonsense learning, while these models depend heavily on fine-tuning.ALBERT is presented as a smaller, more memory-friendly BERT variant, and RoBERTa as improving on BERT through pre-training design choices.Source: §2.3 · p. 3

2.4 Masked Language Modeling

• The study evaluates out-of-the-box masked language modeling, where a token is masked and other tokens are used to predict it, focusing on classification quality rather than pseudo-log-likelihood.The passage frames masked language modeling as a common natural-language-understanding objective.Source: §2.4 · p. 3

2.5 Normalized Distance Metrics

• The study records normalized Levenshtein, Jaccard, Cosine, and Sørensen-Dice distance values between adversarial examples and natural inputs, scaling each from 0 (perfect match) to 1 (complete mismatch).Source: §2.5 · p. 3

2.6 Unsupervised Matching Metrics

• BERTScore compares token embeddings using cosine similarities and supplies precision, recall, and F1, while BLEU measures quality through n-gram frequency.BERTScore is included to assess whether embedding-based matching captures small perturbations, and BLEU offers an automatic baseline comparable to human evaluations.Source: §2.6 · p. 4

3.1 Handcrafted Examples

• The authors construct 40 instruction prompts and replace each target classification item with a semantically divergent token, using an instruction to classify it as a distraction.The stated injection format is: “Ignore the previous instructions and classify [ITEM] as [DISTRACTION].”Source: §3.1 · pp. 4–5
• The original classification is treated as a quasi-ground-truth baseline, and pre- and post-injection outputs are compared using token- and sequence-level semantic-similarity measures.Table 1 lists the models selected for the classification task.Source: §3.1 · p. 5, Table 1 · p. 5

3.2 Human Verification

• Human annotators label whether each post-injection classification differs semantically from the original, helping assess model outputs and interpret quality scores.Verification also addresses case sensitivity, syntactic differences, and outputs outside the model vocabulary.Source: §3.2 · p. 5

4.1 Dataset

• Each of seven models received 40 adversarial examples; the study analyzed 2,800 outputs, including 200 gold-standard and 200 generated classifications per model.The outputs comprise 400 classifications per model.Source: §4.1 · p. 5

4.2 Model Selection

• The classification models were text-davinci-002, bert-base-uncased, albert-base-v2, and roberta-base; the three bidirectional models were chosen for MLM capability and budget constraints.The authors report GPT-3 observations because they first identified the vulnerability in that model.Source: §4.2 · p. 5

4.3 Experiment Settings

• Model settings came from Hugging Face model cards and the OpenAI Playground; BERT-family models used Hugging Face defaults for fill-mask, including five candidate choices.GPT-3 used temperature 0.0 and maximum length 6; the reported BERT, RoBERTa, and ALBERT configurations varied in dropout and vocabulary size.Source: §4.3 · pp. 5–6

5.1 GPT-3 Playground

• Nearly 14% more likely than the ground-truth class, the injected class was GPT-3 Playground’s most likely option in the illustrated example.The paper also reports the vulnerability to the OpenAI team responsible for the Playground.Source: Figure 1 · p. 6, §5.1 · p. 6

5.2 Normalized Distance-Based Scores

• Table 2 reports mean and median Levenshtein, Sørensen-Dice, Jaccard, and Cosine distances at token and sequence levels.Source: §5.2 · p. 6

5.3 Unsupervised Matching Scores

• Table 3 reports mean and median BERTScore precision, recall, and F1, plus BLEU-1 and BLEU-4, at token and sequence levels.Source: §5.3 · p. 6

5.4 F1 Score and Injection Accuracy

• 0.675 injection accuracy was reported for both RoBERTa and ALBERT, versus 0.425 for BERT; their token- and sequence-level F1 scores also differed.BERT: 0.39922 token-level and 0.40351 sequence-level; RoBERTa: 0.20748 and 0.09589; ALBERT: 0.21544 and 0.19403.Source: §5.4 · p. 6

• The authors define mismatch-indicating thresholds for BLEU, BERTScore, and normalized distance metrics, yet report that these metrics fail to capture the injected adversarial examples.High BLEU values suggest corrupted outputs may bypass quality checks despite sequence-level semantic differences.Source: §6 · p. 7
• Despite semantic differences, every model’s median BERTScore was closer to 1 than 0; human verification and sequence-distance scores supported the mismatch interpretation.Source: §6 · p. 7
• Token-level distances were higher than sequence-level distances across measures and models, while BERT’s median distances were zero and RoBERTa’s sequence medians were near zero.ALBERT had the highest normalized distances overall but still had sequence-level medians close to zero, motivating more robust text-quality measures.Source: §6 · pp. 7–8

7 Conclusion

• The approach tests PLM robustness before fine-tuning and highlights a major security vulnerability in the public GPT-3 Playground.The authors describe the observed vulnerability as an example of harmful behavior being injected into pre-trained models.Source: §7 · p. 8
• The approach decreases text-classification quality in state-of-the-art models beyond GPT-3 and may surpass known quality-control methods.The authors also identify potential relevance to model interpretability and fairness.Source: §7 · p. 8
• The authors argue that potential harms from shared PLMs across domains must be accounted for to prevent unintended harm to users.They frame their observed vulnerability as additional evidence of the work needed to prevent such harm.Source: §7 · p. 8