PaperlayerView exact revision on arXiv

Source-linked AI summary

TextBugger: Generating Adversarial Text Against Real-world Applications

Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, Ting Wang

arXiv:1812.05271v1cs.CRcs.CLcs.LG

TL;DR

DLTU systems are increasingly used in security-sensitive text applications, but their security vulnerabilities remain largely unknown. The paper presents TEXTBUGGER, a framework for generating utility-preserving adversarial texts in white-box and black-box settings, and demonstrates successful attacks against real-world systems. The authors also discuss defenses and future improvements to the attack and defense procedures.

  • Problem

    DLTU systems used for applications such as sentiment analysis and toxic content detection have largely unknown security vulnerabilities.

  • Method

    TEXTBUGGER identifies important text units and applies small word or character perturbations to generate utility-preserving adversarial texts under white-box and black-box settings.

  • Results

    100% attack success rate was achieved on IMDB when targeting Amazon AWS and Microsoft Azure under black-box settings.

  • Takeaways & Limitations

    The results demonstrate effective and efficient adversarial attacks against sentiment-analysis and toxic-content-detection models and platforms, while transferred examples indicate vulnerabilities in real applications.

  • Takeaways & Limitations

    The perturbations could be improved with language-processing technologies, beam search, and phrase-level modification, while robust defense schemes remain future work.

Abstract

from arXiv · show

Deep Learning-based Text Understanding (DLTU) is the backbone technique behind various applications, including question answering, machine translation, and text classification. Despite its tremendous popularity, the security vulnerabilities of DLTU are still largely unknown, which is highly concerning given its increasing use in security-sensitive applications such as sentiment analysis and toxic content detection. In this paper, we show that DLTU is inherently vulnerable to adversarial text attacks, in which maliciously crafted texts trigger target DLTU systems and services to misbehave. Specifically, we present TextBugger, a general attack framework for generating adversarial texts. In contrast to prior works, TextBugger differs in significant ways: (i) effective -- it outperforms state-of-the-art attacks in terms of attack success rate; (ii) evasive -- it preserves the utility of benign text, with 94.9\% of the adversarial text correctly recognized by human readers; and (iii) efficient -- it generates adversarial text with computational complexity sub-linear to the text length. We empirically evaluate TextBugger on a set of real-world DLTU systems and services used for sentiment analysis and toxic content detection, demonstrating its effectiveness, evasiveness, and efficiency. For instance, TextBugger achieves 100\% success rate on the IMDB dataset based on Amazon AWS Comprehend within 4.61 seconds and preserves 97\% semantic similarity. We further discuss possible defense mechanisms to mitigate such attack and the adversary's potential countermeasures, which leads to promising directions for further research.

I. INTRODUCTION

The paper examines vulnerabilities in text classification and introduces TEXTBUGGER, which generates utility-preserving adversarial texts that deceive real-world systems under white-box and black-box settings.

  • Deep neural networks used in security-sensitive applications have been found vulnerable to carefully crafted adversarial examples.
  • Text attacks are challenging because text is discrete, small perturbations are perceptible, and word replacements can alter sentence semantics.
  • Existing adversarial-text methods are limited by inefficiency, white-box assumptions, manual intervention, or narrow model coverage.
  • TEXTBUGGER generates utility-preserving adversarial texts against state-of-the-art classifiers in both white-box and black-box settings.
  • 100% attack success rate was achieved on IMDB when targeting Amazon AWS and Microsoft Azure under black-box settings.
  • A user study found that TEXTBUGGER-generated adversarial texts had little impact on human understanding.

II. ATTACK DESIGN

TEXTBUGGER identifies influential text units and applies small, utility-preserving word perturbations, using model gradients in white-box settings and queries or scoring functions in black-box settings.

  • Threat Model: The attack seeks an adversarial document with an incorrect model label while maintaining similarity to the legitimate document.
  • TEXTBUGGER: TEXTBUGGER calls slightly changed words “bugs” and uses them to create adversarial texts in both attack settings.
  • White-box Attack: White-box attacks compute the classifier Jacobian to identify important words, generate five bug types, and select the bug that most reduces ground-truth confidence.
  • Bug Generation: Word-level perturbations use context-aware nearest neighbors to avoid semantic reversals such as replacing “worst” with “better”.
  • Bug Generation: Insert, delete, and swap operations make small character-level changes while preserving recognizable text structure.
  • Black-box Attack: The scoring function is designed to reflect word importance, require no model parameters or architecture, and remain efficient to calculate.

III. ATTACK EVALUATION: SENTIMENT ANALYSIS

The sentiment-analysis evaluation studies TEXTBUGGER’s practical performance using benchmark datasets, targeted models, baselines, and attack metrics.

  • The evaluation investigates adversarial-text generation for sentiment analysis and analyzes results after introducing datasets, models, baselines, metrics, and implementation details.

A. Datasets

The evaluation uses IMDB movie reviews and Rotten Tomatoes movie-review snippets, covering binary sentiment data with substantially different sample lengths.

  • The IMDB dataset contains 50,000 positive and negative movie reviews, split evenly into training and testing sets, with 215.63 words per sample on average.
  • The Rotten Tomatoes dataset contains 5,331 positive and 5,331 negative processed reviews or snippets, averaging 32 words and using an 80%/10%/10% train-validation-test split.

B. Targeted Models

The evaluation targets offline sentiment models and ten online sentiment platforms/models, comparing TEXTBUGGER with baseline attacks and measuring text utility through four similarity and distance metrics.

  • Targeted Models: TEXTBUGGER is evaluated against LR, Kim’s CNN, and an LSTM in white-box attacks.The models are trained with hold-out testing, with hyperparameters tuned only on the validation set.
  • Targeted Models: Black-box evaluation covers ten sentiment analysis platforms and models, including Google Cloud NLP, IBM Watson, Microsoft Azure, Amazon AWS, and fastText.The evaluation also includes ParallelDots, TheySay Sentiment, Aylien Sentiment, TextProcessing, and Mashape Sentiment.
  • Baseline Algorithms: The white-box comparison includes Random, FGSM+Nearest Neighbor Search, and DeepFool+Nearest Neighbor Search.Random modifies 10% of each sentence’s words, while the other baselines generate adversarial embeddings and reconstruct text through nearest-neighbor search.
  • Evaluation Metrics: Utility is measured with edit distance, Jaccard similarity, Euclidean distance, and semantic similarity.The first two metrics operate on raw text, while Euclidean distance and semantic similarity operate on word vectors.
  • Evaluation Metrics: Semantic similarity uses Universal Sentence Encoder sentence vectors and cosine similarity to measure meaning preservation between original and adversarial texts.The attack controls semantic similarity above a specified threshold.

E. Implementation

The experiments report TEXTBUGGER’s attack performance across white-box and black-box settings, including effectiveness on benchmark datasets, sensitivity to document length, and concrete adversarial examples.

  • Attack Performance: TEXTBUGGER achieves high black-box attack success rates and outperforms DeepWordBug across real-world online DLTU platforms.On IMDB, it achieves 100% success against Azure and AWS, compared with 56.3% and 68.1% for DeepWordBug.
  • Attack Performance: TEXTBUGGER succeeds by perturbing only a few words, including about 2 words for one MR sample with 96.8% success against Microsoft Azure.That sample perturbs 7% of its words and comes from documents averaging 32 words.
  • The Impact of Document Length: Document length has little impact on attack success rate, but longer documents weaken confidence changes on IBM Watson and Google Cloud NLP.Generation time generally increases with document length for Microsoft Azure and Google Cloud NLP.
  • Adversarial Text Examples: Six character- or word-level modifications change a CNN prediction from 99.8% negative to 81.0% positive.The example includes insertion, case substitution, deletion, and character substitution operations.
  • Score Distribution: Across all samples, TEXTBUGGER shifts sentiment scores toward the positive direction, including cases where it does not flip the predicted label.The score-change analysis covers both successful and failed attacks.

G. Utility Analysis

TEXTBUGGER generates adversarial texts that preserve substantial word-level and semantic utility across white-box and black-box settings. Its utility remains stable as document length increases because the perturbed-word proportion stays small.

  • Almost 90% of adversarial texts preserve at least 0.9 semantic similarity under white-box attacks.
  • TEXTBUGGER-generated texts are more similar to originals than DeepWordBug outputs at both word and vector levels in black-box attacks.
  • The Impact of Document Length: For IBM Watson and Microsoft Azure, perturbed-word counts roughly increase with document length, whereas Google Cloud NLP shows little change.
  • The Impact of Document Length: Increasing the number of perturbed words does not decrease semantic similarity because TEXTBUGGER controls their proportion within a small range.

H. Discussion

The toxic-content evaluation examines TEXTBUGGER across datasets, models, and online platforms, focusing on important toxic words and perturbation choices. The analysis also considers document length and semantic similarity.

  • Discussion: Important words identified against the CNN model are predominantly negative terms, whose slight modification reduces the input text’s negative extent.Examples include “bad,” “awful,” “stupid,” “worst,” and “terrible.”
  • Discussion: Perturbation choices vary across platforms: insertion dominates Microsoft Azure and Amazon AWS, whereas Sub-C dominates IBM Watson and fastText.Sub-C is designed to create visually similar adversarial texts, while swap, insertion, and deletion resemble common typos.
  • Experimental Setup: The toxic-content evaluation uses the Kaggle Toxic Comment Classification dataset, converting six toxicity categories into binary classification.The dataset contains human-labeled Wikipedia comments.
  • Experimental Setup: The experiments evaluate TEXTBUGGER against self-trained LR, CNN, and LSTM models under white-box settings.Data are split 80%, 10%, and 10% for training, validation, and testing.
  • Experimental Setup: The black-box evaluation targets Google Perspective, IBM Natural Language Classifier, Facebook fastText, ParallelDots AI, and Aylien Offensive Detector.Two models are trained using the Kaggle data because only training and prediction interfaces are available.

C. Attack Performance

TEXTBUGGER achieves strong attack performance on toxic-content models and platforms while changing relatively few words. Its modifications also shift confidence toward non-toxic predictions and preserve textual utility.

  • Effectiveness and Efficiency: TEXTBUGGER perturbs only a few words, achieves high attack success rates, and outperforms baseline algorithms across white-box and black-box evaluations.Randomization has minor influence on the white-box results.
  • Effectiveness and Efficiency: 92.3% success rate is achieved on the LR model by perturbing 10.3% of words, while every baseline remains at or below 40%.With an average Kaggle document length of 55 words, this corresponds to about six perturbed words.
  • Effectiveness and Efficiency: 82.1% attack success rate is achieved on ParallelDots by perturbing 4.0% of words, or about three words per sample.
  • Score Distribution: After modification, the overall confidence distribution shifts toward non-toxic predictions across all evaluated platforms and models.The analysis includes both successful and failed samples.
  • Utility: TEXTBUGGER preserves more utility than DeepWordBug and maintains good word-level similarity under both white-box and black-box settings.Nearly 80% of white-box adversarial texts have no more than 20 edits in one reported comparison.
  • Toxic Words Distribution: Slightly perturbing toxic words such as “fuck” and “dick” decreases the toxic score of toxic content.
  • Bug Distribution: Sub-C is the dominant black-box operation across all platforms, while Sub-W is the least used.The authors report that these results are similar to those in another section.

V. FURTHER ANALYSIS

Further analysis examines transferability, human perception, and the visibility of TEXTBUGGER’s perturbations. The results indicate cross-model and cross-platform attack potential while largely preserving human judgments of sentiment and toxicity.

  • Transferability: Adversarial texts generated for one model show a moderate degree of transferability to other models and platforms.The evaluation covers LR, CNN, and LSTM models across three datasets.
  • Transferability: 54.3% attack success rate is achieved on ParallelDots by adversarial texts generated against the Kaggle LR model.The authors state that transferability could enable attacks on online platforms despite call limits.
  • User Study: The user study samples 500 legitimate and 500 adversarial texts from the IMDB and Kaggle datasets, including white-box and black-box examples.Participants labeled each text as positive/non-toxic or negative/toxic.
  • User Study: 94.9% of adversarial samples are classified by human participants as their original labels, compared with 95.5% of legitimate samples classified correctly.Most incorrect classifications occur on samples with ambiguous expressions.
  • User Study: Participants detect 30.1% of added bugs, compared with 38.0% of pre-existing errors in the sampled texts.Added bugs account for 65.5% of all errors, while pre-existing errors account for 34.5%.

VI. POTENTIAL DEFENSES

The paper evaluates spelling checks and adversarial training as potential defenses against TEXTBUGGER. Spelling checks only partly reduce attack effectiveness, while adversarial training is constrained by its dependence on attack knowledge and examples.

  • Defense strategies: Spelling checks are evaluated under black-box attacks, while adversarial training is evaluated under white-box attacks.The spelling-check service is Microsoft Azure’s context-aware system.
  • Spelling check: 35.6% success rate remains for TEXTBUGGER versus 16.5% for DeepWordBug on Perspective API after spelling correction.The comparison indicates that spelling correction does not eliminate TEXTBUGGER’s effectiveness on that platform.
  • Spelling check: On IMDB, insert bugs are easiest to correct, while on Kaggle, delete bugs are easiest to correct.The correction ratios are analyzed by bug type using Fig. 17.
  • Spelling check: Sub-W bugs have a correction ratio below 10%, making them the hardest bug type to correct.The paper connects this low correction rate to TEXTBUGGER’s stronger performance than DeepWordBug.
  • Adversarial training: Adversarial training may improve robustness, but it requires knowledge of the attack strategy and sufficient adversarial texts for training.Because attackers typically do not disclose these materials, the defense is limited against unknown adversarial attacks.
  • Adversarial countermeasures: Attackers can increase the proportion of Sub-W bugs or adjust perturbation proportions across platforms to improve attack robustness.These strategies are presented as countermeasures to the discussed defenses.

VII. DISCUSSION

The discussion describes TEXTBUGGER’s extensions, distinctions from prior adversarial-text attacks, and defense limitations. It also identifies broader evaluation coverage and future directions for improving perturbations and defenses.

  • Extension to Targeted Attack: TEXTBUGGER can be adapted to targeted attacks by computing the Jacobian with respect to the target label instead of the ground-truth label.The paper’s experiments otherwise perform untargeted attacks.
  • Limitations and Future Work: TEXTBUGGER’s perturbations remain limited because more sophisticated language-processing algorithms could improve them.Suggested techniques include syntactic parsing, named entity recognition, and paraphrasing.
  • Related methods: Prior gradient-based attacks can significantly alter original meaning, while other methods rely on heuristic rules, hand-crafted synonyms, or typos.The discussion contrasts these approaches with the paper’s adversarial-text framework.
  • Related methods: Existing text attacks include out-of-vocabulary substitutions, semantically similar replacements, distracting sentences, and GAN-generated sequences.The cited methods also have practical constraints such as manual intervention or limited text length.
  • Defense gap: Text-domain defenses have not been systematically studied, and prior adversarial-training work focused on clean-example accuracy rather than textual adversarial examples.This motivates the paper’s preliminary defense exploration.
  • Remarks: TEXTBUGGER combines character- and word-level perturbations, emphasizes efficiency, and evaluates attacks on 15 real-world online DLTU systems.The paper reports broader online evaluation than prior work and describes the method as more general and robust.
  • Conclusion: The results indicate potential vulnerabilities in real applications, while spelling checks and adversarial training remain possible defense directions.The paper proposes linguistically or structurally aware ensembles as a further robustness direction.
Loading 1812.05271v1…