Source-linked AI summary
Deep Text Classification Can be Fooled
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, Wenchang Shi
TL;DR
DNN-based text classifiers are vulnerable to adversarial attacks, but text perturbations must remain imperceptible and preserve utility. The paper identifies influential text items through white-box gradients or black-box occlusion, then applies insertion, modification, and removal strategies. The resulting samples fool both character-level and word-level classifiers, while the method’s black-box setting requires training-set access and confidence information, and crafting currently requires human effort.
Problem
DNN-based text classifiers were seriously underestimated in adversarial-sample research, while effective attacks must preserve text meaning, readability, and utility.
Method
The method identifies important text items using cost gradients in white-box attacks or occluded test samples in black-box attacks, then applies insertion, modification, and removal perturbations.
Results
The generated adversarial samples successfully fool both representative character-level and word-level DNN text classifiers while satisfying imperceptible-perturbation and utility-preservation requirements.
Takeaways & Limitations
DNN-based text classification is surprisingly vulnerable to adversarial-sample attack, so its robustness should be seriously considered.
Takeaways & Limitations
The black-box attack assumes access to the training set and confidence degree information, while crafting samples still requires human effort.
Abstract
from arXiv · showhide
In this paper, we present an effective method to craft text adversarial samples, revealing one important yet underestimated fact that DNN-based text classifiers are also prone to adversarial sample attack. Specifically, confronted with different adversarial scenarios, the text items that are important for classification are identified by computing the cost gradients of the input (white-box attack) or generating a series of occluded test samples (black-box attack). Based on these items, we design three perturbation strategies, namely insertion, modification, and removal, to generate adversarial samples. The experiment results show that the adversarial samples generated by our method can successfully fool both state-of-the-art character-level and word-level DNN-based text classifiers. The adversarial samples can be perturbed to any desirable classes without compromising their utilities. At the same time, the introduced perturbation is difficult to be perceived.
1 Introduction
The paper addresses the underestimated vulnerability of DNN-based text classifiers while requiring adversarial samples to remain imperceptible and utility-preserving. It introduces attacks that identify influential text items and perturb them through insertion, modification, or removal.
- Existing adversarial-sample research focused on image and audio classifiers, leaving DNNs for natural-language processing seriously underestimated.
- Text attacks must preserve meaning and human readability because small discrete changes can make text unnatural or meaningless.The paper contrasts text’s discrete structure with the perturbation tolerance of continuous image and audio data.
- The method uses cost gradients for white-box attacks and occluded test samples for black-box attacks to identify what, where, and how to perturb.These signals guide insertion, modification, and removal rather than directly overlapping multimedia-style perturbations with the input.
- Experiments attack representative character-level and word-level DNNs using adversarial samples designed to fool models while preserving utility and limiting perceptibility.The supplied introduction reports successful source/target misclassification attacks against both model types and satisfaction of all three stated requirements.
2 Target Models and Datasets
The evaluation targets representative character-level and word-level DNN text classifiers across named datasets. The character-level model processes one-hot character vectors, while the word-level model uses pretrained word embeddings and convolutional classification.
- The character-level DNN is evaluated on DBpedia, with 560,000 training samples, 70,000 testing samples, and 14 high-level classes.Characters are represented with one-hot vectors before processing through six convolutional and three fully connected layers.
- The word-level model uses 300-dimensional pretrained word embeddings, followed by convolution, max pooling, dropout, a fully connected layer, and softmax output.
- The word-level model is tested on MR, CR, and MPQA, including movie-review and product-review sentiment datasets.MR contains 10,662 reviews, while CR contains 3,775 product reviews categorized as Positive or Negative.
3 White-box Attack
The white-box attack identifies classification-important text items using cost gradients, then manipulates them through insertion, modification, and removal strategies designed to preserve readability and meaning.
- Identifying Classification-important Items: Cost gradients identify characters and phrases that contribute strongly to the current classification.Characters with maximum gradient magnitude are called hot characters, while influential phrases are Hot Sample Phrases (HSPs).
- Identifying Classification-important Items: The attack uses HTPs to construct payloads and HSPs to determine where text should be inserted, modified, or removed.The three strategies can be applied separately or in combination, including targeted source/target misclassification.
- Insertion Strategy: Insertion places HTP-based payloads near relevant HSPs to increase confidence in a desired target class.A single HTP, such as “historic,” can change a company description toward the Building class; multiple HTPs may be assembled into a parenthesis or fact.
- Insertion Strategy: Natural-language watermarking supports stealthy insertion by embedding multiple HTPs in dispensable facts, forged facts, presuppositions, or semantically empty phrases.These additions are intended to preserve the text’s primary semantics while avoiding human attention.
- Modification and Removal Strategies: Modification changes HSPs along cost-gradient directions using common misspellings or visually similar characters, while removal targets supplementary HSP words.Replacing “film” with “flim” follows the desired gradient directions, and removing “British” from an HSP lowers original-class confidence by 35.0%.
- Combination of Three Strategies: Combining insertion, modification, and removal can succeed when each strategy alone only lowers confidence without changing the prediction class.In the cited example, the individual strategies lower confidence by 27.3%, 17.5%, and 10.1%, respectively, while their combination changes the output classification.
4 Black-box Attack
The black-box attack identifies influential text items without access to the target model’s internal knowledge, then uses insertion, modification, and removal to craft adversarial samples. Occlusion testing can identify hot phrases and words, with performance comparable to the white-box approach.
- Black-box identification: The black-box attack probes the target model with purposefully generated test samples to locate hot text phrases and words without internal model knowledge.The method uses fuzzing-inspired testing while keeping test-sample generation consistent across character-level and word-level models.
- Black-box identification: Occluding each word with same-length whitespace preserves text structure while testing its effect on model confidence.Redundant whitespace contributes little to semantics, whereas structural changes can affect some text classifiers.
- Black-box identification: Occluding “television” and “British” produced confidence declines of 42.2% and 26.9%, identifying them as hot phrases in the tested text.The method collects hot words across labeled training samples to identify frequent hot text phrases.
- Black-box identification: Black-box testing identified seven of the top ten Positive hot text phrases and eight of the top ten Negative hot text phrases also found by white-box analysis.The overlap was reported for the CR dataset.
- Adversarial generation: Black-box testing identified the hot items used in the presented examples and supported the same insertion, modification, and removal strategies as white-box attack.The authors report that black-box attack is as effective as the white-box approach.
5 Evaluation
The evaluation tests targeted misclassification, human detectability and utility, efficiency, and agreement between white-box and black-box identification. The method achieved desired target attacks while preserving utility and producing difficult-to-detect perturbations, but required substantial computation for the largest dataset.
- Q1: Misclassification attack: The method crafted adversarial samples that changed a DBpedia sample from Means of Transportation to each of the other 13 classes.Additional samples from the other 13 classes were also perturbed into desirable target classes.
- Q2: Human perception and utility: Human observers detected only 5.0% of introduced modifications by accuracy and 2.0% by recall, suggesting the perturbations were difficult to perceive.There were 12 successful matches among 240 marked locations, while the method produced 594 changes.
- Q3: Efficiency: Computing white-box hot phrases for all DBpedia classes took 116 hours, while black-box identification took 107 hours; crafting one sample took about 15 minutes.Hot phrases for the smaller MR, CR, and MPQA datasets were obtained within several minutes.
- Q4: Attack comparison: White-box and black-box methods produced about 80% overlapping hot text phrases and often identified the same hot phrases for individual samples.The authors describe the two approaches as effective and complementary because each can find items missed by the other.
6 Related Work
Earlier adversarial-sample research developed several attack and defense techniques, primarily for non-text DNN classifiers. Initial text attacks exposed vulnerabilities but faced compatibility or applicability limitations.
- Adversarial samples and defenses: Adversarial-sample generation has used gradient-based, decision-function-based, and evolution-based approaches, with gradient-based methods described as efficient and easy to use.Defensive distillation and adversarial training were discussed as defenses, while robust defense construction remained open.
- Text-specific challenge: Image- and audio-oriented adversarial methods are not directly applicable to text because text is discrete and sensitive to perturbation.The paper contrasts this with the greater perturbation tolerance of continuous data.
- Prior text attacks: Jia and Liang caused reading-comprehension systems to output wrong answers by appending distracting sentences, but those sentences tended to be incompatible with the target paragraph.The passage identifies this as an early text adversarial-sample approach.
7 Discussion and Conclusion
The paper concludes that DNN-based text classification is vulnerable to adversarial attacks and argues that robustness deserves serious consideration. It also identifies practical constraints and directions for refining the attack methods.
- Conclusion: The study reports effective white-box and black-box attacks that craft text adversarial samples across different adversarial scenarios.The conclusion presents vulnerability of DNN-based text classification as an important underestimated finding.
- Limitations and future work: The black-box attack assumes access to the training set and confidence-degree feedback, resources that may not always be available.The authors suggest indirect attacks through a trained substitute model as a possible refinement.
- Limitations and future work: Crafting an adversarial sample currently requires human effort, motivating future work on automatic large-scale generation.The authors consider the effort acceptable for common attack scenarios but identify automation as a future objective.