Source-linked AI summary
Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
Di Jin, Zhijing Jin, Joey Tianyi Zhou, Peter Szolovits
TL;DR
Text data remains difficult to attack while preserving human prediction consistency, semantic similarity, and language fluency. TEXTFOOLER ranks important words and replaces them with semantically similar, grammatically correct alternatives in black-box models; across classification and entailment tasks, it reduced target accuracy below 10% with fewer than 20% of words perturbed while preserving human judgments of correctness, meaning, and grammaticality.
Problem
Natural-language attacks must fool models while preserving human prediction consistency, semantic similarity, and language fluency, but previous methods barely satisfied all three requirements.
Method
TEXTFOOLER identifies important words in black-box target models and prioritizes replacing them with semantically similar and grammatically correct words until the prediction changes.
Results
Almost all target models across five classification and two entailment tasks reached below 10% accuracy with fewer than 20% of original words perturbed, while human evaluations found the examples correct, semantically similar, and grammatical.
Takeaways & Limitations
TEXTFOOLER provides a strong baseline for generating utility-preserving adversarial text against state-of-the-art classification and entailment models in the black-box setting.
Takeaways & Limitations
Generated adversarial samples remain susceptible to word-sense ambiguity, grammatical errors, and task-sensitive content shifts.
Abstract
from arXiv · showhide
Machine learning algorithms are often vulnerable to adversarial examples that have imperceptible alterations from the original counterparts but can fool the state-of-the-art models. It is helpful to evaluate or even improve the robustness of these models by exposing the maliciously crafted adversarial examples. In this paper, we present TextFooler, a simple but strong baseline to generate natural adversarial text. By applying it to two fundamental natural language tasks, text classification and textual entailment, we successfully attacked three target models, including the powerful pre-trained BERT, and the widely used convolutional and recurrent neural networks. We demonstrate the advantages of this framework in three ways: (1) effective---it outperforms state-of-the-art attacks in terms of success rate and perturbation rate, (2) utility-preserving---it preserves semantic content and grammaticality, and remains correctly classified by humans, and (3) efficient---it generates adversarial text with computational complexity linear to the text length. *The code, pre-trained target models, and test examples are available at https://github.com/jind11/TextFooler.
Introduction
TextFooler is introduced as a black-box method for generating natural adversarial text that changes model predictions while preserving human-perceived utility. Across classification and entailment settings, it is evaluated for attack effectiveness, efficiency, and preservation of semantics and grammaticality.
- Motivation: TextFooler targets the challenge of fooling text models while preserving human prediction consistency, semantic similarity, and language fluency.Prior text attacks often failed to satisfy all three utility-preserving requirements.
- Approach: TextFooler identifies influential words and replaces them with semantically similar, grammatically correct alternatives until the target prediction changes.The approach operates without access to model architecture or parameters.
- Results: Almost all target models in all tasks fall below 10% accuracy after attack with less than 20% of the original words perturbed.Human evaluation also found the adversarial examples correctly classified, semantically similar, and grammatically acceptable.
- Evaluation: The framework attacks three state-of-the-art models across five text-classification tasks and two textual-entailment tasks.The evaluated models include convolutional, recurrent, and BERT-based architectures.
- Contributions: The paper contributes a comprehensive automatic and human evaluation framework for assessing attack effectiveness, efficiency, and utility preservation.The authors also release code, target models, and test samples for future benchmarking.
Method
TextFooler formulates adversarial text generation as changing a model prediction while maintaining a minimum similarity to the original sentence. It ranks influential words, filters and tests replacement candidates, and iteratively applies the best valid substitutions.
- Problem Formulation: A valid adversarial example must change the model prediction while keeping similarity to the original text above a threshold.The similarity function is intended to capture semantic and syntactic similarity in natural language.
- Problem Formulation: In the black-box setting, the attacker uses only target-model predictions and confidence scores, without access to architecture, parameters, or training data.This setting motivates a query-based importance-ranking mechanism.
- Step 1: Word Importance Ranking: TextFooler ranks words by the prediction change caused by deleting each word, prioritizing words that most influence the original classification.The score compares the target-label prediction before and after deletion, with a separate case when the predicted label changes.
- Step 2: Word Replacement: For each selected word, the method generates nearby synonym candidates, filters them by part of speech, and checks sentence-level semantic similarity.The candidate pool uses embedding cosine similarity, with N and δ controlling diversity and similarity; the reported setting is N=50 and δ=0.7.
- Step 2: Word Replacement: The algorithm iteratively replaces ranked words, preferring a candidate that changes the prediction with the highest semantic similarity and otherwise the candidate lowering the original-label confidence most.Stop-word filtering is used to help avoid grammatical destruction.
Experiments
The experiments evaluate TextFooler on diverse text-classification and textual-entailment datasets against convolutional, recurrent, InferSent, ESIM, and BERT models. Attack success is assessed through accuracy changes, while human studies examine semantic similarity, grammaticality, and classification consistency.
- Tasks and Datasets: The evaluation covers text classification and textual entailment using 1,000 randomly selected test examples per dataset.The classification datasets span news topics, fake-news detection, and sentence- or document-level sentiment analysis.
- Tasks and Datasets: The classification benchmark includes AG’s News, Fake News, MR, IMDB, and Yelp Polarity, while entailment uses SNLI and MultiNLI.SNLI contains 570K sentence pairs, and MultiNLI covers multiple written and spoken genres.
- Target Models: Target models include WordCNN, WordLSTM, and BERT for classification, plus InferSent, ESIM, and fine-tuned BERT for entailment.The models are trained on the training sets before adversarial examples are generated from test samples.
- Evaluation Protocol: Attack success is measured by comparing original test accuracy with accuracy after attacking the trained target models.A larger gap between original and after-attack accuracy indicates a more successful attack under the paper’s evaluation procedure.
- Human Evaluation: Human evaluation measures semantic similarity, grammaticality, and classification accuracy on shuffled original and adversarial texts.The study samples 100 test sentences for WordLSTM on MR and BERT on SNLI.
Results
TEXTFOOLER substantially reduces target-model accuracy across text classification and entailment while using limited word perturbations. Its adversarial examples also outperform prior attacks and preserve human-perceived grammaticality and semantic content.
- Automatic attack results: TEXTFOOLER reduces state-of-the-art model accuracy to below 15% across five classification and two entailment tasks, generally with less than 20% word perturbation.The stated exceptions are the Fake dataset and AG under BERT.
- Automatic attack results: 5.1% word perturbation reduces WordLSTM accuracy on IMDB from 89.8% to 0.3%, while 3.5% reduces WordCNN accuracy to 0%.IMDB averages 215 words, and successful attacks perturb 10 words or fewer per sample.
- Automatic attack results: BERT accuracy falls about 5–7 times on classification and 9-22 times on NLI, including Yelp from 95.6% to 6.8% and SNLI from 89.4% to 4.0%.The paper reports these reductions despite BERT's higher original accuracy than the non-pretrained comparison models.
- Automatic attack results: Higher original model accuracy generally corresponds to higher after-attack accuracy and perturbed-word ratios, while the Fake dataset remains harder to attack.The authors report that converting real news to fake is easier than the reverse process.
- Benchmark and human evaluation: TEXTFOOLER beats prior published systems on both attack success rate and perturbed word ratio, while human ratings find adversarial text near-original in grammaticality and meaning.Human label agreement is 92% on MR and 85% on SNLI; sentence similarity scores are 0.91 and 0.86, respectively.
Discussion
The analysis identifies mechanisms that strengthen TextFooler attacks while preserving utility, and documents transferability, robustness gains from adversarial training, and remaining error modes.
- Ablation analysis: Removing word-importance ranking raises after-attack accuracy by more than 45% on MR, AG, and SNLI, making the attack ineffective.The ranking step locates words with the greatest effect on target predictions and reduces the number of perturbed words.
- Ablation analysis: Removing the semantic similarity constraint eases attacks but lowers semantic similarity, revealing a trade-off between attack effectiveness and meaning preservation.The constraint filters noisy embedding-based synonyms by requiring sentence-level similarity above threshold ϵ.
- Transferability: Adversarial examples show moderate transferability across models, with higher transferability in textual entailment and when generated against BERT.Transferability is evaluated by measuring another model’s accuracy on examples that fooled the source model.
- Adversarial training: Adversarial training increases after-attack accuracy and the perturbed-word ratio for BERT on MR and SNLI, indicating greater attack difficulty.The expanded training data combines original examples with adversarial examples sharing their labels.
- Caveat: Semantic-similarity scores decrease less on NLI datasets because their sentences are generally shorter than 20 tokens and USE is less sensitive to meaning changes there.
- Error analysis: Remaining errors include word-sense ambiguity, grammatical mistakes, and task-sensitive content shifts, especially when substitutions alter textual entailment relations.A replacement that preserves sentiment may nevertheless change entailment, such as replacing “a kid” with “a girl.”
Related Work
Adversarial attacks are well established for continuous image inputs, whereas discrete text requires different approaches because perturbations are more challenging to construct.
- Computer-vision attacks commonly use gradient-based perturbations in continuous input spaces, while adversarial attacks on discrete text are more challenging.
Conclusion
The paper studies black-box adversarial attacks against state-of-the-art text classification and textual entailment models using TEXTFOOLER. Human studies found the generated texts legible, grammatical, and similar in meaning to the originals.
- TEXTFOOLER generates targeted adversarial texts against state-of-the-art text classification and textual entailment models in the black-box setting.
- Human studies validated that the generated adversarial texts are legible, grammatical, and similar in meaning to the original texts.