Source-linked AI summary
Reevaluating Adversarial Examples in Natural Language
John X. Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, Yanjun Qi
TL;DR
Text lacks a shared standard for what makes an adversarial example successful, and inconsistent evaluations hinder comparisons. The paper defines success through model fooling plus linguistic constraints, then evaluates synonym-substitution attacks and tighter constraints. Stricter constraints improve semantic quality but sharply reduce attack success, while matched constraint enforcement clarifies differences between search methods.
Problem
Text has no clear analog to image-style indistinguishable perturbations, and inconsistent definitions and evaluations hinder using adversarial examples to understand and improve NLP models.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Holding constraint application constant showed that TEXTFOOLER’s reported improvement over GENETICATTACK came from lenient enforcement rather than a better search method.
Source: §9 · p. 9The proposed constraint set is not necessarily optimal for every attack scenario; some contexts may need fewer or less strict constraints.
Source: §7 · p. 9
Abstract
from arXiv · showhide
State-of-the-art attacks on NLP models lack a shared definition of a what constitutes a successful attack. We distill ideas from past work into a unified framework: a successful natural language adversarial example is a perturbation that fools the model and follows some linguistic constraints. We then analyze the outputs of two state-of-the-art synonym substitution attacks. We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors. Human surveys reveal that to successfully preserve semantics, we need to significantly increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences.With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.
1 Introduction
Text lacks an image-like notion of indistinguishable perturbations, while inconsistent definitions and evaluations make attack comparisons difficult. The paper proposes a shared framework, evaluates synonym-substitution attacks, and finds that stricter constraints improve example quality at a substantial cost to attack success.
1 Introduction
- The proposed definition treats a successful natural-language adversarial example as one that fools the model and fulfills linguistic constraints.The framework identifies semantics, grammaticality, overlap, and non-suspicion as constraint categories.Source: §1 · p. 1
- Human surveys found that two state-of-the-art synonym-substitution attacks often fail semantic and non-suspicion constraints, while perturbed examples contain 39% more grammar-checker errors than originals.The grammar checker detected error types humans almost never make.Source: §1 · p. 2
- TFADJUSTED tightens semantic-similarity thresholds and checks grammar during search, improving judged semantic preservation and noticeability but lowering attack success from over 80% to under 20%.It uses TEXTFOOLER’s search process; in adversarial training, TEXTFOOLER’s examples decreased model accuracy, but TFADJUSTED’s did not.Source: §1 · p. 2
- With constraint application held constant, GENETICATTACK’s search is more successful than TEXTFOOLER’s, clarifying a comparison previously confounded by differing constraint enforcement.The paper attributes TEXTFOOLER’s reported improvement over GENETICATTACK to lenient constraint application rather than a better search method.Source: §1 · p. 2
2 Constraints on Adversarial Examples in Natural Language
The section defines adversarial success as fooling a model while meeting chosen constraints, and organizes natural-language constraints into four categories. Their relevance depends on the attack scenario, including the desired semantic, grammatical, overlap, and human-perception properties.
2 Constraints on Adversarial Examples in Natural Language
- Adversarial attacks seek an input that meets the attack goal while satisfying each selected constraint; the framework categorizes constraints for natural-language perturbations.The goal may vary by task, while the constraint set depends on the use case.Source: §2 · pp. 2–3
2.1 Semantics
- Semantic constraints require preserving input meaning, although unchanged semantics need not preserve the correct label in every task.The paper illustrates this boundary with Modern versus Early Modern English classification.Source: §2.1 · p. 3
2.2 Grammaticality
- Grammaticality constraints restrict the grammar of the perturbed input, which can matter even when grammatical errors do not change meaning.The paper gives plagiarism evasion as a setting where grammatical correctness matters.Source: §2.2 · p. 3
2.3 Overlap
- Overlap constraints limit character-level similarity changes, using measures such as edit distance or n-gram metrics.A maximum edit distance can suit attacks willing to introduce misspellings.Source: §2.3 · p. 3
2.4 Non-suspicion
- Non-suspicion constraints require perturbed text to appear unmodified, but what seems suspicious depends on context and may not matter in scenarios such as illegal PDF distribution.The paper contrasts a kindergartner’s homework with an academic paper as contexts with different expectations.Source: §2.4 · p. 3
4 Constraint Evaluation Methods and Case Study
The section proposes human and automatic evaluations for linguistic constraints and applies them to synonym-substitution attacks. In the case study, generated examples often fail semantic, grammaticality, and non-suspicion checks.
4 Constraint Evaluation Methods and Case Study
- The case study evaluates GENETICATTACK and TEXTFOOLER, which generate adversarial examples by substituting words with synonyms until the classifier is fooled.The evaluation discusses semantics, grammaticality, and non-suspicion; overlap is omitted because it is easy to evaluate automatically.Source: §4 · p. 4
4.1.1 Human Evaluation
- Semantic evaluation combines human judgments of meaning preservation with automatic similarity proxies, including sentence-encoder cosine similarity and counter-fitted word-embedding neighbors.The paper proposes a 1–5 human rating and notes that thresholds for automatic metrics require human evaluation.Source: §4.1.1 · p. 4, §4.1.2 · p. 5
4.1.3 Case Study
- Human judges rated TEXTFOOLER perturbations 3.28 on average and GENETICATTACK perturbations 2.70, both significantly below the proposed semantic-preservation threshold ϵsem of 4, suggesting neither method was judged semantics-preserving on average.Source: §4.1.3 · p. 5
4.2.1 Human Evaluation
- The section recommends LanguageTool for automatic grammaticality checks and asks human judges whether perturbations introduced errors.For non-suspicion, judges classify shuffled real and perturbed inputs as real or computer-altered; automatic classifiers can also serve as proxies.Source: §4.2.1 · p. 6, §4.2.2 · p. 6, §4.3.1 · p. 6
4.2.3 Case Study
- LanguageTool detected more grammatical errors in perturbed inputs for 50% of TEXTFOOLER examples and 32% of GENETICATTACK examples.Six error categories appeared at least 10 times more frequently in perturbed than original samples.Source: §4.2.3 · p. 6
4.3.3 Case Study
- Judges identified TEXTFOOLER perturbations as computer-altered with 69.2% accuracy, above the 50% expected if all examples were non-suspicious.The study presented judges with a shuffled mix of real and perturbed examples.Source: §4.3.3 · p. 6
5 Producing Higher Quality Adversarial Examples
The authors tighten semantic and grammaticality constraints in TFADJUSTED, which uses TEXTFOOLER’s search process. The stricter constraints improve measured linguistic quality but sharply reduce attack success.
5.1 With Adjusted Constraint Application
- Human judges moved from “Not sure” on TEXTFOOLER semantic preservation to “Agree” for TFADJUSTED.The semantic constraints use minimum cosine similarities for swapped-word embeddings and sentence encodings.Source: §5 · pp. 6–7, §5.1 · p. 7
- LanguageTool checks every TFADJUSTED perturbation, so no perturbation can introduce grammatical errors under the checker.For MR, lowercasing and tokenization make some inputs difficult for LanguageTool to parse.Source: §5.1 · p. 7
- Judges identified TFADJUSTED inputs as computer-altered with 58.8% accuracy, over 10% lower than for TEXTFOOLER examples.The non-suspicion study was repeated with TFADJUSTED examples.Source: §5.1 · p. 7
- Attack success decreased by at least 71 percentage points on each of the three datasets after constraint adjustment.Table 5 reports results for TEXTFOOLER before and TFADJUSTED after adjustment.Source: §5.1 · p. 7, Table 5 · p. 8
5.2 Adversarial Training With Higher Quality Examples
- On MR test accuracy, TEXTFOOLER augmentation decreased accuracy while TFADJUSTED augmentation had a small positive effect.The training comparison used 7,382 TEXTFOOLER examples versus 825 TFADJUSTED examples, averaged over five random seeds.Source: §5.2 · p. 7, Figure 2 · p. 8
5.3 Ablation of TFADJUSTED Constraints
- Removing the word-embedding distance constraint most increased attack success, followed by removing the sentence-encoder constraint.The ablation removed one TFADJUSTED constraint at a time.Source: §5.3 · p. 8, Table 6 · p. 8
6 Comparing Search Methods
The authors compare GENETICATTACK’s genetic algorithm with TEXTFOOLER’s greedy search while holding constraint application constant. GENETICATTACK is more successful, but requires over 40 times more model queries.
6 Comparing Search Methods
- With constraint application held constant, GENETICATTACK’s genetic algorithm was more successful than TEXTFOOLER’s greedy search at both tested constraint levels.The comparison attributes TEXTFOOLER’s higher reported success over GENETICATTACK to more lenient constraint application, not the search method.Source: §6 · p. 8
- GENETICATTACK’s genetic algorithm required over 40x more model queries than TEXTFOOLER’s greedy search.Source: §6 · p. 8
7 Discussion
Tightening semantic constraints improved human-rated example quality but sharply reduced attack success. The discussion also emphasizes that results depend on constraint settings and that attack search should be evaluated separately from constraint application.
7 Discussion
- More selective semantic constraints produced examples rated above 4 for semantic preservation, but attack success dropped steeply.The authors conclude that NLP models are relatively robust to current synonym-substitution attacks when perturbations must preserve semantics and grammaticality.Source: §7 · pp. 8–9
- GENETICATTACK’s genetic algorithm was more successful than TEXTFOOLER’s greedy strategy, but much less efficient.The comparison used 1,000 samples against BERT fine-tuned on MR.Source: Table 7 · p. 9
- Constraint settings may need to vary by context, since some scenarios may require fewer constraints or less strict enforcement.Source: §7 · p. 9
- Researchers should separate search-method changes from constraint changes, or observed attack gains will not identify which change mattered.Source: §7 · p. 9
- Paraphrase attacks with rigorous semantic and grammatical preservation remain an open test of BERT’s robustness.The authors also point to improved paraphrasing and semantic-similarity measurement as ways to explore beyond synonym substitution.Source: §7 · p. 9
8 Related Work
Prior NLP attacks used differing definitions of success, making comparisons difficult. The paper frames adversarial examples through a goal function and linguistic constraints, while situating this framework alongside earlier constraint and evaluation work.
8 Related Work
- Earlier NLP attack work used differing definitions of adversarial-example success, complicating comparisons between methods.Source: §8 · p. 9
- The paper defines an adversarial example using a goal function and a set of linguistic constraints.Source: §8 · p. 9
- Earlier frameworks addressed potential attack-space constraints or meaning preservation, but did not provide the same shared categorization for NLP attacks.Gilmer et al. (2018) did not focus on NLP attacks, while Michel et al. (2019) restricted their framework to sequence-to-sequence models.Source: §8 · p. 9
9 Conclusion
The paper finds that two state-of-the-art synonym-substitution attacks frequently violate the constraints they claim to follow. TFADJUSTED produces examples judged to preserve semantics and grammaticality, and controlled comparisons attribute TEXTFOOLER’s attack-success improvement to more lenient constraint application rather than a better search method.
9 Conclusion
- TEXTFOOLER and GENETICATTACK frequently violated the constraints they claimed to follow.Source: §9 · p. 9
- TFADJUSTED applied constraints that produced adversarial examples judged to preserve semantics and grammaticality.Source: §9 · p. 9
- Holding constraint application constant showed that TEXTFOOLER’s higher attack success came from lenient constraints, not a better search method.The paper argues that shared definitions and application methods can clarify comparisons of attack performance.Source: §9 · p. 9
A.1 Experimental Setup
Experiments used BERT-base and standard dataset splits, with a special train-test split for MR. The authors also make selected samples and survey results available in their GitHub repository.
A.1 Experimental Setup
- Experiments used BERT-base, a 110-million-parameter model, on machines with four TITAN X Pascal GPUs running CentOS 7.Source: §A.1 · p. 12
- The experiments used standard train-test splits except for MR, which was split into 90% training data and 10% testing data.Source: §A.1 · p. 12
- Selected dataset samples and Mechanical Turk survey results were available in the authors’ GitHub repository.Source: §A.1 · p. 12
A.2 Details about Human Studies.
The appendix details the human studies and threshold choices, then reports further analyses of attack detectability, adversarial training, word-embedding distances, and example perturbations.
A.2 Details about Human Studies.
- Human-study workers were recruited from specified countries and experience thresholds, with 100 successful examples sampled per attack-method–dataset combination and 10 responses per example.The study used five datasets; the authors also note a future plan to qualify workers for the task.Source: §A.2 · p. 12
A.2.1 Evaluating Adversarial Examples
- Semantic-similarity judgments compared original and perturbed texts side by side, while the real-versus-altered task was restricted to Movie Review because its samples were shorter.The semantic-similarity prompt asked whether changes preserved the original meaning; each pair received judgments from 10 judges, with average modal-option counts of 5.09 for Section 4 examples and 5.6 for Section 5 examples at threshold .98.Source: §A.2.1 · p. 12
- Synonym-attack examples can appear similar because shared words and small morphological distance influence ratings, so future surveys would filter poor labelers using easy ground-truth items.The authors also studied word-pair replacement judgments.Source: §A.2.1 · p. 12
A.2.2 Finding The Right Thresholds
- The authors chose 0.9 as the word-level cosine-similarity threshold based on participants’ judgments of whether replacing one word preserved sentence meaning.Figure 3 plots average responses by cosine-similarity bins.Source: §A.2.2 · p. 12, Figure 3 · p. 13
A.3 Further Analysis of Non-Suspicious Constraint Case Study
- 62.2% of the time workers guessed perturbed examples were real, while guesses of computer-altered examples were correct 75.4% of the time.The results are presented as a confusion matrix in Table 8.Source: Table 8 · p. 13, §A.3 · p. 13
A.4 Adversarial Training Robustness Results
- The authors observed substantial robustness variance across initializations and epochs, leaving the effects across architectures and datasets unresolved.They trained five models on each dataset and note that it remains to be seen whether examples from GENETICATTACK, TEXTFOOLER, and TFADJUSTED help or hurt robustness and accuracy elsewhere.Source: §A.4 · p. 13
A.5 Word Embeddings
- For normalized word vectors, minimum cosine similarity and maximum Euclidean distance impose equivalent neighbor filters.A minimum cosine threshold ϵ corresponds to maximum Euclidean distance √(2 − 2ϵ).Source: §A.5 · p. 14
A.6 Examples In The Wild
- In eight of ten sampled MR attacks, adjusted constraints caused the attack to fail to find a successful perturbation meeting those constraints.The examples compare original inputs with perturbations under Jin et al. (2019)’s settings and settings adjusted to match human judgments.Source: §A.6 · p. 14, Table 9 · p. 15