Source-linked AI summary
Generating Natural Language Adversarial Examples
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, Kai-Wei Chang
TL;DR
The paper addresses whether natural-language adversarial examples can fool models despite perceptible, discrete word changes. It uses black-box population-based optimization to generate semantically and syntactically similar attacks, achieving success on sentiment analysis and textual entailment while remaining similar to the originals in human evaluation.
Problem
Natural-language adversarial examples are difficult to construct because word changes are perceptible, tokens are discrete, and gradients cannot be computed directly with respect to input words.
Method
The paper uses a black-box, population-based genetic algorithm to minimize word modifications while preserving semantic similarity and syntactic coherence.
Results
The attack succeeds on both sentiment analysis and textual entailment, including a 70% success rate on textual entailment and 92.3% agreement with original sentiment labels among human responses.
Takeaways & Limitations
Semantically and syntactically similar adversarial examples can cause high-performing models to misclassify while humans correctly classify them, motivating robustness research in natural language.
Takeaways & Limitations
The threat model assumes black-box access limited to querying predictions and confidence scores, without knowledge of the model architecture, parameters, or training data.
Abstract
from arXiv · showhide
Deep neural networks (DNNs) are vulnerable to adversarial examples, perturbations to correctly classified examples which can cause the model to misclassify. In the image domain, these perturbations are often virtually indistinguishable to human perception, causing humans and state-of-the-art models to disagree. However, in the natural language domain, small perturbations are clearly perceptible, and the replacement of a single word can drastically alter the semantics of the document. Given these challenges, we use a black-box population-based optimization algorithm to generate semantically and syntactically similar adversarial examples that fool well-trained sentiment analysis and textual entailment models with success rates of 97% and 70%, respectively. We additionally demonstrate that 92.3% of the successful sentiment analysis adversarial examples are classified to their original label by 20 human annotators, and that the examples are perceptibly quite similar. Finally, we discuss an attempt to use adversarial training as a defense, but fail to yield improvement, demonstrating the strength and diversity of our adversarial examples. We hope our findings encourage researchers to pursue improving the robustness of DNNs in the natural language domain.
1 Introduction
The paper extends adversarial-example research to natural language, where semantically and syntactically similar perturbations can cause models to misclassify while humans retain the original judgment. It uses a black-box population-based attack across sentiment analysis and textual entailment, and adversarial training fails to provide robustness.
- Natural-language adversarial examples can be generated against sentiment analysis and textual entailment models using a black-box population-based optimization algorithm.The generated examples are designed to remain semantically and syntactically similar to the originals.
- The attack targets both the IMDB sentiment analysis task and the SNLI textual entailment task.
- Adversarial training failed to yield robustness against the generated attacks, indicating their strength and diversity.
- The attack minimizes semantic and syntactic dissimilarity so humans classify perturbed examples correctly while high-performing models misclassify them.
2 Natural Language Adversarial Examples
Image-domain adversarial methods rely on imperceptible perturbations and continuous, differentiable inputs, assumptions that do not transfer directly to natural language. The paper therefore focuses on semantics-preserving word replacements for natural-language attacks.
- Image attacks typically optimize misclassification while minimizing perceptual distortion, and many popular methods use gradients.
- Imperceptible pixel changes do not transfer to language because word changes are perceptible and words are discrete tokens.
- Word embeddings do not fully solve the problem because nearby embedding replacements can still be noticeable to people.
- Prior natural-language work added distracting sentences, whereas this paper generates semantically and syntactically similar examples through word replacements.
3 Attack Design
The attack uses gradient-free genetic optimization under a black-box threat model, selecting word replacements that preserve meaning and syntax while increasing the target model's prediction score. Its population-based search combines fitness-based selection, crossover, and mutation.
- Threat model: The attacker can query predictions and confidence scores but lacks access to the target model's architecture, parameters, and training data.
- Attack design: The algorithm minimizes modified words while requiring semantic similarity and syntactic coherence, using population-based gradient-free genetic optimization.
- Attack design: Gradient-free optimization enables attacks when model internals are inaccessible and gradients cannot be used.
- Attack design: Genetic algorithms evolve candidate populations through fitness evaluation, probability-weighted parent selection, crossover, and mutation.
- Perturb subroutine: Perturb selects a word and replaces it with a context-fitting semantic neighbor that increases the target-label prediction score.
- Optimization procedure: The optimization initializes a population of distinct modifications and iteratively produces children by combining and perturbing selected parents.
4 Experiments
Experiments evaluate black-box attacks on sentiment analysis and textual entailment models, comparing genetic optimization with a greedy baseline and assessing human perceptibility and adversarial training. The attack achieves high success with limited modifications, while defense experiments provide no additional robustness benefit.
- Experimental setup: The experiments train victim models for IMDB sentiment analysis and SNLI textual entailment.The sentiment model uses an LSTM, while the entailment model predicts entailment, contradiction, or neutrality.
- Experimental setup: The evaluation samples 1,000 sentiment and 500 textual-entailment examples that the victim models classified correctly.The attacker reverses sentiment labels and changes only the hypothesis for textual entailment.
- Attack evaluation: The genetic attack achieves high success rates with limited word modifications and outperforms the Perturb baseline on both tasks.The comparison measures attack success rate and mean percentage of modified words.
- Attack evaluation: 70% textual-entailment success reflects the difficulty of perturbing short SNLI hypotheses, which average 9 words.The IMDB documents average 229 words and were limited to 100 words in the experiments.
- Adversarial training: Adversarial training added no robustness benefit despite near-100% accuracy on adversarial examples included in training.The result is presented as evidence of diverse perturbations and difficulty defending against the attack.
5 Conclusion
The paper concludes that semantically and syntactically similar adversarial examples can be generated for natural-language models using black-box population-based optimization. Human evaluation found the examples adversarial yet perceptibly similar, motivating further robustness research.
- Black-box population-based optimization crafts semantically and syntactically similar adversarial examples for sentiment analysis and textual entailment.
- Human evaluation validated that the generated examples were adversarial and perceptibly quite similar.
- The authors hope the findings encourage research on improving natural-language DNN robustness.
Additional Sentiment Analysis Results
Additional sentiment-analysis attack examples are presented in Table 4, with modified words highlighted separately in the original and adversarial texts.
- Table 4 contains additional attack results for the sentiment-analysis model.
- The table presents examples rather than aggregate sentiment-analysis metrics.
- Modified words are highlighted in green for original texts and red for adversarial texts.
Additional Textual Entailment Results
Additional textual-entailment attack examples are presented in Table 5, with modified words highlighted separately in the original and adversarial texts.
- Table 5 contains additional attack results for the textual-entailment model.
- The table presents examples rather than aggregate textual-entailment metrics.
- Modified words are highlighted in green for original texts and red for adversarial texts.