PaperlayerView exact revision on arXiv

Source-linked AI summary

Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers

Ji Gao, Jack Lanchantin, Mary Lou Soffa, Yanjun Qi

arXiv:1801.04354v5cs.CLcs.CRcs.IRcs.LG

TL;DR

DeepWordBug targets the limited attention given to realistic black-box adversarial attacks on text classifiers. It ranks critical tokens, applies simple character-level transformations, and evaluates the resulting small perturbations across eight datasets. The method reduces average classification accuracy by 68% for Word-LSTM and 48% for Char-CNN models.

  • Problem

    Prior text-adversarial-sample techniques focused largely on white-box attacks, although black-box attacks are more realistic for deployed systems.

  • Method

    DeepWordBug scores important tokens and applies simple character-level transformations to generate small-edit adversarial text sequences in a black-box setting.

  • Results

    68% average decrease in original classification accuracy for Word-LSTM and 48% average decrease for Char-CNN models.

  • Takeaways & Limitations

    DeepWordBug achieves better performance than baseline methods on eight NLP datasets across Word-LSTM and Char-CNN architectures while constraining edit-distance differences.

  • Takeaways & Limitations

    Edit distance does not guarantee that generated sentences retain cogent or coherent technical meaning.

Abstract

from arXiv · show

Although various techniques have been proposed to generate adversarial samples for white-box attacks on text, little attention has been paid to black-box attacks, which are more realistic scenarios. In this paper, we present a novel algorithm, DeepWordBug, to effectively generate small text perturbations in a black-box setting that forces a deep-learning classifier to misclassify a text input. We employ novel scoring strategies to identify the critical tokens that, if modified, cause the classifier to make an incorrect prediction. Simple character-level transformations are applied to the highest-ranked tokens in order to minimize the edit distance of the perturbation, yet change the original classification. We evaluated DeepWordBug on eight real-world text datasets, including text classification, sentiment analysis, and spam detection. We compare the result of DeepWordBug with two baselines: Random (Black-box) and Gradient (White-box). Our experimental results indicate that DeepWordBug reduces the prediction accuracy of current state-of-the-art deep-learning models, including a decrease of 68\% on average for a Word-LSTM model and 48\% on average for a Char-CNN model.

1 INTRODUCTION

DeepWordBug addresses the difficulty of crafting small, realistic adversarial modifications for discrete text in realistic black-box settings. It ranks important tokens and applies simple character-level edits to fool deep classifiers while keeping sequences similar to their originals.

  • Motivation: Deep learning systems used for NLP and security-sensitive applications can be fooled by small test-input modifications that produce incorrect classifications.The paper frames this as a robustness concern for applications such as text-based spam detection.
  • Black-box Setting: Black-box attacks query a classifier without access to its learned model details or input feature representations, making them more realistic for cloud-deployed systems.The Perspective API illustrates this setting because its model runs on cloud servers and exposes no structure or parameters.
  • Problem: Text adversarial perturbations are difficult to define because text is symbolic and discrete, while image attacks can use continuous-space Lp-norm distances.The paper therefore motivates edit distance as a text-oriented measure of modification.
  • Approach: DeepWordBug ranks important tokens and modifies them with small character-level edits so adversarial sequences remain similar to the originals while changing the classifier’s decision.The paper gives an example in which changing only a few characters changes positive sentiment to negative sentiment.
  • Contributions: DeepWordBug operates in a black-box setting and is evaluated on eight real-world text classification tasks against Random and Gradient baselines.The authors report that it fools two deep RNN models more successfully than the state-of-the-art baselines.
  • Implications: The proposed techniques may help reveal vulnerabilities in deep neural networks applied to other discrete inputs such as malware.This is presented as a broader implication of the text-adversarial-sequence techniques.

2 DEEPWORDBUG

The paper represents each example as a token sequence paired with a class label and models classification as a mapping from samples to labels.

  • Data Representation: Each input is represented as a sequence x = x_1x_2x_3...x_n containing n tokens.A token may be a word or a character, depending on the model.
  • Labels: Each label set y contains one of K classes, written as {1, ..., K}.
  • Classifier: A machine-learning model is represented as F : X → Y, mapping the input set X to the label set Y.

2.1 Background

The section introduces RNNs and CNNs as neural architectures used for language inputs, alongside word- and character-level representations that convert text into model features.

  • Recurrent Models: RNNs use recurrent structures to capture sequential dependencies and can model variable-length inputs for NLP tasks.The passage lists sentiment analysis, parsing, and translation as applications.
  • Convolutional Models: CNNs use sliding filters to extract connectivity patterns and have also achieved state-of-the-art performance on language inputs.
  • Word Representations: Word embeddings map dictionary words to vectors, transforming discrete word representations into continuous features for machine-learning models.
  • Character Representations: Character-based models treat each character as a token and can achieve state-of-the-art results when trained on large datasets.

2.2 Method: Black-box Generation of Adversarial Sequences

The paper targets black-box, untargeted attacks where an adversary cannot access a classifier’s structure, parameters, or gradients. It uses a two-step process to identify important tokens and modify them slightly to evade classification.

  • Black-box attacks query the target classifier without access to its structure, parameters, or gradients.
  • The method addresses gradient-based attacks’ limitations on symbolic text inputs and unavailable model parameters.
  • DeepWordBug first ranks tokens by their importance to the classifier’s decision.
  • It then applies slight, imperceptible modifications to selected tokens to create adversarial samples.

2.3 Step 1: Token Scoring Function and Ranking

The first step ranks tokens by measuring their effect on classifier predictions without requiring model parameters or architecture. Four scoring functions capture token importance through replacement or sequential context.

  • The scoring functions estimate token importance from classifier outputs rather than model parameters or architecture.
  • Replace-1 Score: Replace-1 Score measures the prediction change caused by replacing a token with an alternative token.
  • Replace-1 Score: Unknown-token replacement provides a consistent way to compare token importance when searching possible replacements is impractical.
  • Temporal Head Score: Temporal Head Score measures the difference between predictions after reading through token i and token i −1.
  • Temporal Head Score: Temporal Head Scores for all tokens can be computed with a single forward pass in RNN models.
  • Temporal Tail Score: Temporal Tail Score evaluates a token’s influence by comparing trailing sequences that include or omit it.
  • Combined Score: Combined Score integrates head and tail importance to assess a token using its surrounding context.
  • After scoring, the method selects the top m tokens for perturbation.

2.4 Step 2: Token Transformer

The second step transforms highly ranked tokens with small character-level edits, favoring misspellings that remain visually or morphologically similar while altering classifier inputs.

  • The transformer modifies the top m important tokens after token scoring, with word-based transformations requiring a larger search space.
  • Because black-box attacks lack gradients, the method deliberately creates misspelled words using few character modifications.
  • Misspellings can map important words to an unknown dictionary entry whose embedding differs from the original word’s embedding.
  • Levenshtein distance limits perturbation size to preserve human readability while tolerating a defined number of typos.
  • The four transformations are swapping adjacent letters, substituting a letter, deleting a letter, and inserting a letter.
  • Substitution, deletion, and insertion have edit distance 1, whereas swapping has edit distance 2.

3 EXPERIMENTS ON EFFECTIVENESS OF ADVERSARIAL SEQUENCES

The experiments evaluate DeepWordBug across eight datasets and two deep-learning model types, comparing its token-scoring strategies with Random and Gradient baselines. With edit distance limited to 30, DeepWordBug substantially reduces accuracy, transfers across models, and is more sensitive to scoring than transformer choice.

  • Experimental setup: The evaluation covers eight datasets, including text classification, sentiment analysis, and spam detection, using Word-LSTM and Char-CNN target models.
  • Main results: DeepWordBug reduces Word-LSTM accuracy by 68% and Char-CNN accuracy by 48% on average when at most 30 characters are modified.These results outperform the Random and Gradient baselines in the reported comparison.
  • Main results: For Word-LSTM, accuracy falls from 90% to around 25% on AG’s News and from 95% to around 36% on Amazon Review Polarity.The configuration uses Combined Scoring with the Substitution Transformer and ϵ = 30.
  • Perturbation size: On the Enron Spam Dataset, changing 16% of Word-LSTM input words produces an average 73% accuracy decrease.The average is based on changing 30 of 190 words.
  • Robustness across model types: DeepWordBug evades both word-level and character-level classifiers, while randomly selecting tokens has little influence on the final result.The findings support the importance of token selection for the attack.
  • Transferability: Adversarial samples generated for one model reduce another model’s accuracy from around 90% to 20–50%, including models with different word embeddings.Most adversarial samples successfully transfer to other models.
  • Configuration robustness: Changing token-scoring functions affects attack results more than changing token transformers, while dictionary size has only a small influence.Swapping performs worse when its edit-distance difference is 2 rather than 1.

4 CONNECTING TO PREVIOUS STUDIES

Earlier adversarial-text attacks often rely on heuristic modifications, word statistics, embeddings, or white-box gradients. DeepWordBug instead targets realistic black-box settings with edit-distance search and simpler token modifications.

  • Earlier attacks: Good Word attacks add positive, non-spam words to spam messages but lack guarantees for deep models and sample quality.Adding too many words can change the message so it is no longer a useful false-positive attack.
  • Earlier attacks: Image adversarial-example research established small perturbations as a way to fool deep classifiers, whereas text received comparatively little attention.Gradient-based methods subsequently modified NLP inputs for RNN classifiers.
  • Earlier attacks: Prior text methods use embedding gradients, class-word frequencies, synonyms, typos, or heuristic rules to identify and modify important words.The cited gradient-based technique is a white-box attack because it accesses model gradients.
  • DeepWordBug’s distinction: DeepWordBug does not require the target model’s structure, parameters, or gradient, making it applicable in a black-box setting.This contrasts with the paper’s characterization of most previous approaches.
  • DeepWordBug’s distinction: DeepWordBug searches for perturbations in sequence-level edit distance and uses simpler token modifications than the closely connected prior approach.

5 DISCUSSIONS AND ANALYSES

The analyses examine readability, transferability, class and token biases, confidence, adversarial training, and autocorrection. They show that attack behavior depends on model representation and token bias, while training and preprocessing can alter its effectiveness.

  • Sample properties: DeepWordBug produces samples differing from their seeds by only a few letter-level modifications while remaining understandable to humans with the same meaning.The paper acknowledges that readability is subjective.
  • Transferability: Transferability enables a local-model black-box strategy, but DeepWordBug reportedly outperforms the more-informed gradient baseline in that setting.
  • Class bias: Attack effectiveness varies by class: Word-LSTM accuracy falls to 20% for negative samples but remains 60% for positive samples.The experiment uses the Combined Score.
  • Token bias: In Enron Spam, 70% of occurrences of the “unknown” token are in spam emails, biasing attacks that map words to “unknown” toward spam predictions.
  • Token bias: SPACE is less biased than “unknown” in the character-level model because it occurs frequently in both spam and non-spam emails.
  • Prediction confidence: With ϵ = 30, 90% of generated Enron adversarial samples make the model predict incorrectly with confidence above 0.9.This indicates that the wrong predictions are not merely low-confidence outputs.
  • Adversarial training: Adversarial training raises accuracy on adversarial samples from around 12% to 62% after 10 epochs, while slightly decreasing accuracy on normal samples.The training set combines generated adversarial samples with their corresponding originals.
  • Autocorrection: A spellchecker reduces attack performance, but removing two characters from each selected word still lowers four-class accuracy to 34%.

6 CONCLUSION

DeepWordBug is a black-box attack framework that generates adversarial text sequences and outperforms baselines across eight NLP datasets and two deep-learning architectures. Its generated samples also transfer across models, remain effective across dictionary sizes, and can improve adversarial-sample accuracy when used in training.

  • DeepWordBug generates adversarial sequences in a pure black-box manner and performs better than baseline methods across eight NLP datasets using Word-LSTM and Char-CNN architectures.
  • 68% average accuracy decrease occurs for the Word-LSTM model, compared with 48% for the Char-CNN model.
  • Adversarial samples generated on one model transfer to other models, reducing target-model accuracy from around 90% to 20–50%.
  • The number of words in the input dictionary does not affect DeepWordBug’s result.
  • 12% to around 62%: training with DeepWordBug samples increases model accuracy on generated adversarial samples.

7 APPENDIX

The appendix examines attack behavior across transformations, dictionary sizes, confidence, dataset bias, and sample classes. It reports broadly stable performance across transformations and dictionary sizes, while showing effects from edit distance, confidence, class, and dataset composition.

  • Figure 14 compares baseline and DeepWordBug token-scoring methods on Char-CNN models across eight datasets.The X axis is maximum allowed character-level edit distance, and the Y axis is accuracy on generated adversarial samples.
  • Transformer functions: Varying transformation functions has little influence on attack performance, while swapping is worse because it costs twice the edit distance.
  • Dictionary size: DeepWordBug works with dictionary sizes ranging from 5,000 to 20,000 words across all eight datasets.
  • Figure 15 compares baseline and DeepWordBug token-scoring methods on Word-LSTM models across eight datasets.The X axis is maximum edit distance measured by modified words, and the Y axis is adversarial-sample accuracy.
  • Confidence: At least half of the adversarial samples on every dataset mislead the model to a probability 0.2 above a random guess.The experiment uses combined scoring, the substitute transformer, and maximum edit distance difference ϵ = 30.
  • Figures 16–21 organize results by transformation, dictionary size, confidence, dataset bias, and sample class using edit distance and model accuracy.
  • Dataset bias: Word-frequency bias varies substantially across datasets, while character-level data is less biased than word-level data.Unknown-word bias is large in some datasets, such as Enron Spam, and small in others, such as AG’s News.
  • Sample classes: DeepWordBug’s attack performance differs across sample classes, a pattern that may be caused by dataset bias.
Loading 1801.04354v5…