Source-linked AI summary
Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems
Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, Iryna Gurevych
TL;DR
The paper asks whether visually altered text creates an adversarial weakness for NLP systems despite remaining understandable to humans. It introduces visual attacks and shielding methods, finding substantial model sensitivity and improved—but still incomplete—robustness from shielding.
Problem
Current NLP systems generally lack visual character similarity, leaving a gap between human processing and machine processing that can be exploited through visually altered text.
Method
The paper introduces VIPER, evaluates visual attacks across character-, word-, and sentence-level tasks, and tests visual character embeddings, adversarial training, and rule-based recovery.
Results
Performance drops by up to 82% under visual attacks, while the three shielding methods substantially improve model robustness.
Takeaways & Limitations
Humans reliably recover visually perturbed text, whereas NLP models require richer visual biases and further shielding research to narrow this gap.
Takeaways & Limitations
Shielding methods still fall behind non-attack performance, indicating that visual attacks remain difficult to handle.
Abstract
from arXiv · showhide
Visual modifications to text are often used to obfuscate offensive comments in social media (e.g., "!d10t") or as a writing style ("1337" in "leet speak"), among other scenarios. We consider this as a new type of adversarial attack in NLP, a setting to which humans are very robust, as our experiments with both simple and more difficult visual input perturbations demonstrate. We then investigate the impact of visual adversarial attacks on current NLP systems on character-, word-, and sentence-level tasks, showing that both neural and non-neural models are, in contrast to humans, extremely sensitive to such attacks, suffering performance decreases of up to 82\%. We then explore three shielding methods---visual character embeddings, adversarial training, and rule-based recovery---which substantially improve the robustness of the models. However, the shielding methods still fall behind performances achieved in non-attack scenarios, which demonstrates the difficulty of dealing with visual attacks.
1 Introduction
The paper frames visually altered text as an adversarial blind spot: humans can recover its meaning, while NLP systems may fail. It introduces VIPER and evaluates shielding methods against these attacks.
- Visual similarity influences how humans assess character meaning, including in Internet slang, abusive comments, logos, and domain spoofing.
- Current neural NLP systems generally lack built-in visual character similarity and instead treat characters as discrete units or learned embeddings.
- Visually perturbing text can exploit this human–machine processing gap while remaining recoverable by humans, including in malicious emails and toxic comments.
- Visual attacks exchange characters with visually similar alternatives and require neither broad linguistic knowledge nor access to model parameters or loss functions.
- The study investigates SOTA model sensitivity to visual attacks and evaluates shielding with visual character embeddings, adversarial training, and rule-based recovery.
- Performance drops by up to 82% under VIPER attacks, while humans are only mildly or not at all affected by visual perturbations.
2 Related Work
Prior work studied adversarial manipulation and visual character representations separately. This paper combines visually similar character substitutions with NLP attacks designed to preserve human readability.
- Adversarial attacks modify classifier inputs to induce incorrect decisions while preserving the original meaning for human observers.
- Earlier NLP attacks include character flipping, irrelevant paragraphs, question-term replacement, and other noise, often targeting specific model weaknesses.
- Unlike character-level attack methods, the paper allows perturbing any character in a word while potentially minimizing impairment for humans.
- Adversarial training and rule-based corrections are established responses, while many other shielding approaches originated in computer vision.
- Visual character embeddings previously supported tasks such as author and publisher identification in compositional languages.
- Existing visual character-embedding work had not used visual information to attack NLP systems or shield them from such attacks.
3 Approach
The approach uses VIPER to replace characters with visually similar neighbors and compares visual character spaces for attacking and shielding NLP models. VELMo supplies fixed image-based character representations during training.
- Text perturbations: VIPER randomly replaces input characters with visual nearest neighbors from a selected character embedding space.
- Text perturbations: Each character is independently flipped with probability p; replacements are sampled from up to 20 nearest neighbors in the chosen embedding space.
- Text perturbations: VIPER is a black-box attacker that requires no knowledge of the attacked system, though targeting task-specific hot words would make attacks less generally applicable.
- Character Embeddings: ICES provides continuous 576-dimensional image embeddings, whereas DCES and ECES are discrete spaces used to specify VIPER neighbors.
- Character Embeddings: ECES uses manually selected diacritic substitutions for the 52 letters, modeling minimal human-perception change with potentially large NLP impact.
- Character Embeddings: DCES and ECES attack test data, while ICES supplies character representations for deep learning systems; ICES is not used for attacks.
- Word Embeddings: VELMo replaces learned character embeddings with fixed image-based embeddings so ELMo can use visual input features during training.
4 Human annotation experiment
The human annotation experiment tests recovery of original English sentences from visually perturbed text. Humans remain highly accurate across perturbation types and levels, with DCES easier to parse than ICES and ECES nearly perfectly robust.
- Evaluation: The error rate was normalized edit distance, interpreted as the average fraction of characters recovered incorrectly.Distances were averaged across sequence pairs and annotators and normalized by the maximum sequence length.
- Results: Error rates increased consistently with perturbation level for both DCES and ICES, but humans found DCES easier to parse than ICES.DCES preserves a variant of the same character, whereas ICES can change characters, numbers, and punctuation into visually different symbols.
- Results: At least 93% of characters were correctly recovered on average in every condition, even at the 80% disturbance level.Humans also correctly recovered 75% of easy sentences exactly, compared with 72.5% in clean.
- Results: Humans were almost perfectly robust to easy ECES perturbations, making ECES and DCES suitable attack spaces for testing NLP systems.The easy condition merged perturbation levels p = 0.4 and p = 0.8 because their results were nearly identical.
5 Computational Experiments
The experiments evaluate VIPER's impact across character-, word-, and sentence-level NLP tasks, then compare four shielding approaches. Visual perturbations substantially degrade all systems, while shielding improves robustness but does not consistently restore clean-data performance.
- Tasks: The evaluation covers G2P, POS tagging, chunking, and toxic-comment classification across character-, word-, and sentence-level tasks.G2P is character-level; POS tagging and chunking are word-level; toxic-comment classification is sentence-level.
- Systems: The study compares neural and non-neural systems, including Marmot, the Stanford POS tagger, FastText, and word-level dependency embeddings.Marmot is feature-based, while the Stanford tagger primarily uses lexical features.
- VIPER attacks: All systems degrade considerably under DCES perturbations: at 40% disturbed characters, all three POS taggers retain below 60% of their clean score.Chunking degrades more strongly, while toxic-comment classification degrades least among the reported tasks.
- VIPER attacks: G2P loses 40% of performance at 10% perturbation, whereas toxic-comment classification loses 30% only at p = 0.9.The authors suggest character-level tasks are more affected than sentence-level tasks, while noting metric and task-specific lower bounds.
- Human-machine contrast: DCES and ECES produce virtually identical degradation curves for machines, unlike humans, for whom ECES is much easier to parse.This contrast highlights a discrepancy between human and machine processing of visual perturbations.
- Shielding: Adversarial training, visual character embeddings, their combination, and rule-based recovery improve robustness, but their effectiveness varies by task and attack setting.AT+CE is substantially better than either method alone at p = 0.5 for POS, while RBR performs especially well on ECES.
- Shielding: At p = 0.9, AT+CE-shielded chunking remains below 60% of clean performance, partly because unseen attacks induce domain shift.The authors also identify non-optimal visual character embeddings as a possible contributing factor.
6 Discussion
The discussion examines robustness gains from adversarial training and shielding, alongside attack behavior in toxic-comment classification. Visual attacks can selectively target offensive words, while visually informed training is more robust than random perturbation training.
- Adversarial training: Random noise improved robustness, but remained less effective than visually informed adversarial training.The comparison is reported for POS and toxic-comment classification examples.
- Error analysis: 24.1% of toxic comments received a lower toxicity level after VIPER with DCES at p = 0.1.The attack affected roughly one fourth of toxic comments, while toxicity increased in only 3.2% of non-toxic comments.
- Error analysis: Perturbing hot words substantially reduced toxicity level, whereas perturbing non-hot words such as “he” had little effect.In one example, a nonshielded model’s toxicity level fell from 3.29 to 0.11.
- Shielding: Shielding with AT+CE was more robust to stronger attacks than rule-based recovery in the showcased examples.The comparison concerns attacks with higher visual dissimilarity.
- Error analysis: 55% of successful attacks perturbed at least one word from a list of 20 hand-selected offensive words.The authors suggest that the small list may underestimate the effect of perturbing offensive words.
7 Conclusion
The conclusion presents visual text modification as an adversarial threat that exposes a fundamental gap between human and machine processing. It argues that richer model biases and further shielding research are needed, while highlighting applications in moderation and privacy-preserving communication.
- 7 Conclusion: Humans reliably recover visually perturbed text, whereas state-of-the-art NLP systems can fail substantially under visual attack.The experiments span character-, word-, and sentence-level tasks.
- 7 Conclusion: The authors call for models with richer biases to bridge information-processing gaps between humans and machines.Visual encoding is presented as one such bias, not a complete solution.
- 7 Conclusion: Further work on shielding remains necessary because the explored visual encoding does not fully close the human–machine processing gap.The conclusion explicitly identifies shielding as an ongoing research need.
- 7 Conclusion: The findings matter for deployed toxic-comment detectors facing visual attacks and for users seeking privacy or censorship avoidance.VIPER may suggest visually similar spellings for online communication.
- 7 Conclusion: The work extends NLP brittleness beyond machine translation, word reordering, and ordinary word replacements.The authors hope this broader view encourages more ubiquitous shielding techniques.
A.1 SELMo and VELMo Hyperparameters
The appendix specifies deviations from the original ELMo configuration used in the experiments, including smaller convolutional, recurrent, projection, and training settings.
- ELMo configuration: CNN filters of sizes 6 and 7 were excluded relative to the original ELMo configuration.
- ELMo configuration: The maximum characters per token was set to 20 instead of 50.
- ELMo configuration: LSTM dimensionality was set to 2048 instead of 4096, and projection dimensionality to 256 instead of 512.
- ELMo configuration: Models were trained for 5 epochs instead of 10 epochs.
A.2 Task Settings
The appendix describes task-specific datasets, model settings, evaluation metrics, and experimental procedures for G2P, POS, Chunking, toxic-comment classification, and VIPER perturbation analysis.
- G2P: G2P uses the Combilex American English pronunciation dataset with random train/dev/test splits and edit distance averaged over 1k test strings.Results are averaged over 5 random initializations.
- Attack evaluation: Figure 5 plots degradation of state-of-the-art systems across perturbation levels under VIPER(p,ECES).Colored regions compare the performance of other state-of-the-art systems with the authors’ systems.
- POS & Chunking: POS and Chunking use the CoNLL-2000 splits with a readily available LSTM-CRF tagger adapted for ELMo-type input embeddings.The experiments use 10 runs, reporting accuracy for POS and micro-F1 for Chunking.
- Toxic comment classification: Toxic-comment classification uses organizer-provided train/test splits, a 10k-sentence development set, averaged ELMo word embeddings, and a tuned MLP.Models are trained on clean and perturbed data using SELMo and VELMo representations.
A.3 ECES Results
On ECES-perturbed data, the paper evaluates system degradation and compares adversarial training, character embeddings, and rule-based recovery as shielding methods.
- ECES results: RBR recovers ECES data almost perfectly across perturbation levels.ECES changes each character only to a single, extremely similar nearest neighbor, allowing RBR to almost completely undo the perturbations.
- ECES results: Figure 6 compares AT (ICES) and CE on ECES-perturbed data, with AT (Random) shown as colored regions.The figure’s ∆AT y axis spans −0.15–0.3 for visualization.
- ECES results: Figure 7 compares AT+CE (ICES) and RBR on ECES-perturbed data, with AT (Random) shown as colored regions.
A.4 AT+CE vs. AT or CE
AT+CE is evaluated against AT and CE, alongside intrinsic comparisons of VELMo and SELMo sentence embeddings under visual perturbations. The combined shielding method generally performs best, while VELMo better preserves similarity under stronger perturbations.
- AT+CE comparison: AT+CE almost constantly outperforms either AT or CE, especially on word- and sentence-level tasks.The comparison normalizes differences in performance decrease by clean-test performance.
- Intrinsic evaluation: The intrinsic evaluation compares sentence embeddings from clean, perturbed, and randomly chosen sentences using cosine similarity.Sentence embeddings are formed by averaging VELMo or SELMo word embeddings.
- Intrinsic evaluation: With larger perturbation probabilities, VELMo’s similarity ratio with the source sentence increases from 70% to 95% for ICES.This indicates that VELMo better captures similarity to the source sentence under stronger perturbation.
- Intrinsic evaluation: The VELMo advantage over SELMo is consistent but can be small, possibly because contextual information lets SELMo infer characters from word and sentence context.
- Intrinsic evaluation: When all characters are perturbed, SELMo reaches 1% while VELMo reaches more than 34–39% for similarity to the original sentence.Thus, VELMo embeddings better capture similarity between visually similar words.