PaperlayerView exact revision on arXiv

Source-linked AI summary

Bad Characters: Imperceptible NLP Attacks

Nicholas Boucher, Ilia Shumailov, Ross Anderson, Nicolas Papernot

arXiv:2106.09898v2cs.CLcs.CRcs.LG

TL;DR

Text-based NLP systems can be affected by imperceptible encoding changes, but the supplied passages describe these risks without detailing prior attacks’ limits. The paper explores black-box encoding-based attacks across NLP tasks and reports broad vulnerability among tested models and systems. It recommends defenses while noting that application needs and resource constraints can complicate their use.

  • Problem

    Text-based NLP models can be altered by imperceptible language-coding features, which may also increase inference runtime.

    Source: §VIII · p. 13

  • Method

    The paper explores black-box attacks using encoding perturbations such as invisible characters, homoglyphs, reorderings, and deletions.

    Source: §I · p. 2

  • Results

    Across eight models, task-appropriate benchmark performance fell to near zero in untargeted attacks; all tested systems were vulnerable to three variants, and most to four.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors recommend defenses for text-based NLP systems, while noting that application requirements and resource constraints may limit available options.

    Source: §VIII · p. 13

Abstract

from arXiv · show

Several years of research have shown that machine-learning systems are vulnerable to adversarial examples, both in theory and in practice. Until now, such attacks have primarily targeted visual models, exploiting the gap between human and machine perception. Although text-based models have also been attacked with adversarial examples, such attacks struggled to preserve semantic meaning and indistinguishability. In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs. We use encoding-specific perturbations that are imperceptible to the human eye to manipulate the outputs of a wide range of Natural Language Processing (NLP) systems from neural machine-translation pipelines to web search engines. We find that with a single imperceptible encoding injection -- representing one invisible character, homoglyph, reordering, or deletion -- an attacker can significantly reduce the performance of vulnerable models, and with three injections most models can be functionally broken. Our attacks work against currently-deployed commercial systems, including those produced by Microsoft and Google, in addition to open source models published by Facebook, IBM, and HuggingFace. This novel series of attacks presents a significant threat to many language processing systems: an attacker can affect systems in a targeted manner without any assumptions about the underlying model. We conclude that text-based NLP systems require careful input sanitization, just like conventional applications, and that given such systems are now being deployed rapidly at scale, the urgent attention of architects and operators is required.

I. INTRODUCTION

The paper presents black-box attacks that use imperceptible encoding changes to affect text-based NLP systems. It describes their potential to hinder content moderation and search indexing, and discusses defenses.

I. INTRODUCTION

  • Four black-box variants target integrity and availability using imperceptible encoding perturbations, including invisible characters, homoglyphs, reorderings, and deletions.Source: §I · p. 2
  • Across eight models, task-appropriate benchmark performance fell to near zero in untargeted attacks; all tested systems were vulnerable to three variants, and most to four.Source: §I · p. 2
  • Most targeted attacks succeeded, while sponge example attacks slowed inference by at least a factor of two.Source: §I · p. 2
  • The evaluation spans eight models and tasks including translation, toxic-content detection, entailment, named entity recognition, and sentiment analysis.Source: §I · p. 2
  • The paper presents defenses, while noting that application requirements and resource constraints may prevent some defenses from being used.Source: §I · p. 2

II. MOTIVATION

  • Earlier NLP attacks were noticeable or changed meaning, whereas this class aims to preserve semantic meaning while remaining imperceptible.Source: §II · p. 2
  • Imperceptible perturbations could stifle machine translation and toxic-content detection of inflammatory sentences, and make malicious content harder to find through search indexing.Source: §II · p. 2

A. Adversarial Examples

Adversarial examples manipulate model outputs, and NLP attacks have been difficult to make both semantically faithful and visually imperceptible. The paper situates its approach against prior NLP attacks and describes the language-model and Unicode representations relevant to those attacks.

A. Adversarial Examples

  • Adversarial examples cause incorrect inference outputs; white-box attacks can optimize loss, while black-box attacks may transfer examples or approximate gradients from observed outputs.Adversarial algorithms can also target model availability and confidentiality, as well as integrity.Source: A. Adversarial Examples · pp. 2–3

B. NLP Models

  • Transformers map source-language sequences to target-language sequences through an encoder representation and a target-language decoder.NLP systems process human language; machine translation has evolved from rule-based and statistical approaches to neural models, including RNNs and Transformers.Source: B. NLP Models · p. 3
  • Subword encoding can enable translation of previously unseen words, whereas word dictionaries have limited vocabularies and n-gram dictionaries grow exponentially.N-gram encoding still does not solve the unseen-word problem.Source: B. NLP Models · p. 3

C. Adversarial NLP

  • Prior attacks used synonym and character swaps, paraphrases [12], or embedded-space neighbors; noisy inputs hurt translation and classification [9],,, while misspellings create visible artifacts.Source: C. Adversarial NLP · p. 3
  • The paper’s attacks use fewer substitutions, introduce no visible perturbations, and preserve semantic meaning, unlike prior techniques that produce visual artifacts.The paper also presents a taxonomy of adversarial NLP attacks.Source: C. Adversarial NLP · p. 4

D. Unicode

  • Unicode assigns characters numerical code points, which can be encoded as one to four UTF-8 bytes.Fonts map code points to glyphs, and unsupported code points typically render as an unknown-character placeholder.Source: D. Unicode · p. 4

E. Unicode Security

Unicode security issues include homoglyphs and controls that let displayed character order differ from encoded order. The paper focuses on attacks that exploit the gap between visualization and NLP pipelines, using four classes of imperceptible perturbation.

E. Unicode Security

  • Unicode’s complexity can create security issues, and implementation bugs have produced varied exploits across different software ecosystems.The paper distinguishes these bugs from its focus on attacks exploiting correct implementations.Source: E. Unicode Security · pp. 4–5
  • Unicode homoglyphs are distinct characters with the same or nearly the same glyph, and character sequences can also create pseudo-homoglyphs such as “rn” and “m”.Homoglyphs have also been used in scams and information hiding; URL protections such as Punycode do not address cross-script homoglyph vulnerabilities outside URLs.Source: E. Unicode Security · p. 4
  • Unicode bidirectional controls can make characters render in a different order than they are encoded, and overrides have been used to disguise file extensions.Source: E. Unicode Security · p. 4

A. Attack Taxonomy

  • These attacks exploit the gap between visualization and NLP pipelines, while the paper sets aside bugs and focuses on correct Unicode implementations.The taxonomy notes that other text-attack classes produce visual artifacts.Source: E. Unicode Security · p. 5, A. Attack Taxonomy · p. 5
  • Four attack classes alter model inputs through invisible characters, homoglyphs, reorderings, or deletions while changing the rendering not at all or only subtly.The taxonomy is abstracted from particular encoding standards; experiments use Unicode, with generalization proposed for encodings with sufficiently large character and control-sequence sets.Source: A. Attack Taxonomy · p. 5
  • Imperceptible perturbations can manipulate machine translation, toxic-content classifiers, search querying and indexing, and sponge-example behavior for denial-of-service attacks.Source: A. Attack Taxonomy · p. 5

B. NLP Pipeline

NLP pipelines process text through tokenization and encoding, while rendering systems handle characters differently; the paper exploits this gap with black-box encoding perturbations. Invisible characters remain valid encoded input and can alter model outputs despite producing no rendered glyph.

B. NLP Pipeline

  • NLP pipelines tokenize and encode text, while rendering engines process a richer set of control characters, creating the representational gap the attacks exploit.Evaluated open-source models use preprocessing such as Byte Pair Encoding or WordPiece before dictionary lookup.Source: B. NLP Pipeline · p. 5

C. Attack Methodology

  • The black-box attack optimizes imperceptible perturbations using differential evolution, with integrity or availability goals and a budget limiting input distance.Integrity attacks change the output or match a target; availability attacks increase inference time.Source: C. Attack Methodology · pp. 5–6

D. Invisible Characters

  • Invisible characters render without a glyph or taking space, but their visibility is font-specific in practice.The paper notes that fonts may render traditionally invisible characters visibly.Source: D. Invisible Characters · p. 6
  • Invisible characters can alter model output, degrading accuracy and runtime when injected arbitrarily and enabling desired output changes when injected in a targeted fashion.These characters remain valid encoded input despite producing no rendered glyph.Source: D. Invisible Characters · p. 7

E. Homoglyphs

Homoglyph attacks use distinct Unicode characters that render alike or similarly. The paper grounds its experimental mappings in Unicode technical reports while noting that rendering and homoglyph identification can depend on the font.

E. Homoglyphs

  • Homoglyphs are distinct characters that render as the same or visually similar glyph, and their appearance can vary by font.For the experiments, the paper uses Unicode technical reports to define homoglyph mappings.Source: E. Homoglyphs · p. 7
  • The Unicode technical reports describe mappings for characters intended to share a glyph and for characters likely to be visually confused.The paper uses the official mappings for reproducibility, while noting that unsupervised clustering can produce similar results.Source: E. Homoglyphs · p. 7

F. Reorderings

Bidi controls can change encoded character order without changing rendered text, while deletion controls can remove neighboring text from view. These perturbations alter what NLP models process and degrade performance.

F. Reorderings

  • Figure 4 plots Unicode Security Confusables homoglyph clusters as a 2D PCA of Unifont glyph images processed with VGG16.The accompanying discussion says rendered glyphs are transformed into embeddings for clustering.Source: Figure 4 · p. 7
  • Bidi control characters can shuffle encoded character order without affecting rendering, though implementation differences can make some reorderings platform- or application-specific.The paper describes reorderings as font-independent and reports testing its algorithm against Chromium’s Unicode implementation.Source: F. Reorderings · p. 8

G. Deletions

  • Deletion controls such as backspace, delete, and carriage return can remove rendered text; attacks are generally platform-independent but may require direct submission of encoded bytes.Deleted text is usually not copied to the clipboard, limiting reliance on a victim’s copy-and-paste action.Source: G. Deletions · p. 8

A. Integrity Attack

  • Invisible characters, homoglyphs, reorderings, and deletions disrupt model inputs in different ways, and each modification degrades performance through model-specific effects.The paper describes unknown tokens, word splitting, less-common embeddings, encoded-order processing, and processing of deleted characters as possible mechanisms.Source: A. Integrity Attack · p. 8

B. Availability Attack

The paper examines targeted sponge examples alongside integrity attacks, using black-box tests across NLP systems. Its machine-translation integrity results include perturbation-related BLEU comparisons and approximately linear Levenshtein-distance increases, with reorderings producing the largest distances.

B. Availability Attack

  • Sponge examples can target inference runtime using homoglyphs at fixed input size or invisible-character injections when increased input size is acceptable.The paper notes that parallel attacks at scale may degrade hosted-model availability.Source: B. Availability Attack · p. 8

A. Experiment Setup

  • Experiments cover four perturbation classes, five NLP tasks, and eight open- and closed-source models in a black-box setting without model-weight or state access.The genetic algorithm used a population of 32 and at most 10 iterations; the authors describe results as a lower bound because larger settings might find more effective perturbations.Source: A. Experiment Setup · p. 9

B. Machine Translation: Integrity

C. Machine Translation: Availability

The experiments report runtime slowdowns and transferable translation attacks, as well as degradation in toxic-content detection and untargeted textual entailment. In textual entailment, performance drops significantly even at perturbation budget 1.

C. Machine Translation: Availability

  • Reordering attacks were the most effective sponge examples against Fairseq English-French translation, although their slowdown was smaller than that reported by Shumailov et al. [18].The authors state these attacks remain semantically meaningful and visually unnoticeable.Source: C. Machine Translation: Availability · p. 10

D. Machine Translation: MLaaS

  • Imperceptible adversarial examples generated against Google Translate and Microsoft Azure ML appeared effective when transferred between the two services.The cross-service results are plotted in the appendix figures cited by the authors.Source: D. Machine Translation: MLaaS · p. 10

E. Toxic Content Detection

  • Homoglyphs, reorderings, and deletions degraded IBM Toxic Content Classifier performance by up to 75%, while invisible characters had no effect.The authors suggest invisible characters may have been present in training data or disregarded by the tokenizer.Source: E. Toxic Content Detection · p. 10

F. Toxic Content Detection: MLaaS

G. Textual Entailment: Untargeted

H. Textual Entailment: Targeted

Targeted imperceptible attacks achieved high success rates across textual entailment, named entity recognition, and sentiment analysis, though invisible characters had no effect on the latter two models. At budget 10, imperceptible perturbations had efficacy similar to prior perceptible methods.

H. Textual Entailment: Targeted

  • Targeted textual-entailment attacks reached 80.0% success at budget 5, while label-only attacks reached 79.6%.The label-only attacks had access to the selected output label rather than class probabilities, and their disadvantage diminished as budgets increased.Source: H. Textual Entailment: Targeted · p. 11, Figure 11 · p. 11

I. Named Entity Recognition: Targeted

  • Targeted NER attacks reached 90.2% success at budget 5 depending on technique, but invisible characters had no effect.Success meant at least one output token received the target label.Source: §I · p. 11

J. Sentiment Analysis: Targeted

  • Targeted sentiment-analysis attacks reached 79.2% success at budget 5 depending on technique, but invisible characters had no effect.The experiment used a DistilBERT model fine-tuned on the Emotion dataset.Source: J. Sentiment Analysis: Targeted · p. 11

K. Comparison with Previous Work

  • At budget 10, imperceptible perturbations had adversarial efficacy similar to the compared perceptible methods, without visual artifacts.The authors state the imperceptible budget could be increased without visual effect for better adversarial performance.Source: K. Comparison with Previous Work · p. 12

A. Ethics

The experiments show that imperceptible perturbations degrade NLP performance as their budget increases, with a stated exception for invisible-character attacks on three task types. The section also describes attacks on search queries and indexing, and reports the study’s ethics safeguards.

A. Ethics

  • The study used legitimate, well-formed third-party API calls, paid for commercial products, limited inputs and iterations to reduce service and environmental impact, and followed responsible disclosure processes.For example, despite Azure allowing inputs of 10,000 characters, the researchers used inputs shorter than 50 characters.Source: A. Ethics · p. 12

B. Experimental Interpretation

  • Performance metrics and inference time degraded relative to no perturbations, with degradation growing as the perturbation budget increased; invisible-character attacks had no effect on toxic-content, NER, and sentiment models.The authors suggest the exception may reflect invisible characters in training data or tokenizers ignoring the chosen characters.Source: B. Experimental Interpretation · p. 12

C. Search Engine Attack

  • Search-query perturbations interfere with comparisons against indexed resources, while indexing perturbations can make crawled documents less likely to appear in searches using unperturbed terms.The paper gives hiding negative information in financial filings from specialist search engines as an example.Source: C. Search Engine Attack · p. 12

D. Attack Potential

  • The attacks can alter machine-translation outputs, evade toxic-content detection, poison training sets, hide documents from indexing, degrade search quality, and enable denial-of-service attacks.The authors state that all tested text-based NLP systems are susceptible and that any model ingesting user-supplied text is theoretically vulnerable.Source: D. Attack Potential · p. 12

E. Defenses

The paper presents preprocessing and character-specific defenses against imperceptible encoding attacks, while noting that OCR has accuracy and computational costs. Its conclusion recommends defenses for text-based NLP systems that seek robustness against malicious actors.

E. Defenses

  • Rendering inputs and passing them through OCR fully prevented invisible-character, reordering, and deletion attacks, and strongly mitigated most homoglyph attacks, but lowered baseline BLEU by 6.2%.The authors attribute the cost to OCR errors, including more frequent misinterpretation of homoglyphs; better OCR may reduce it.Source: E. Defenses · p. 13

1) Invisible Character Defenses:

  • Removing a standard set of invisible characters before inference blocks invisible-character attacks in many applications; if removal is unsuitable, tokenizers must include them in the source-language dictionary.The paper notes that invisible characters generally concern formatting rather than semantic meaning.Source: 1) Invisible Character Defenses · p. 13

2) Homoglyph Defenses:

  • A model-designer mapping from less-common homoglyphs to common counterparts could approximate a high-performing OCR model, but creating a comprehensive mapping is daunting because Unicode is immense.Automated techniques may help create these mappings.Source: 2) Homoglyph Defenses · p. 13

3) Reordering Defenses:

  • Resolving bidirectional control characters and coercing logical text order to match visual order offers a general defense, including for applications without a graphical user interface.For some GUI-based models, stripping such characters as input is displayed or warning about them may be suitable.Source: 3) Reordering Defenses · p. 13

4) Deletion Defenses:

  • Deletion characters can be resolved before inference or trigger a warning, since the authors suspect few use cases require them as valid model input.Source: 4) Deletion Defenses · p. 13

VIII. CONCLUSION

  • The conclusion proposes varied defenses and recommends that firms deploying text-based NLP systems implement them to improve robustness against malicious actors.The attacks are described as affecting NLP systems across multiple tasks and search and indexing applications.Source: §VIII · p. 13

APPENDIX

The appendix identifies visualizations of edit distances, BLEU scores, and targeted attack success rates, and gives algorithm descriptions for Unicode reorderings and OCR preprocessing.

A. Machine Translation Fairseq Levenshtein Distances

  • Figure 14 concerns Levenshtein distances for integrity-attack perturbations versus unperturbed WMT data on Fairseq EN-FR.Source: Figure 14 · p. 16

B. Example BLEU Scores

  • Figure 15 plots Levenshtein distances for availability attacks, while Table III covers BLEU scores across invisible-character budgets for one Fairseq WMT14 EN-FR input and reference translation.Source: Figure 15 · p. 16, Table III · p. 16

C. Machine Translation MLaaS Results

  • Figures 16 and 18 cover Azure adversarial-example BLEU scores on Google Translate and Levenshtein distances on Google Translate’s EN-FR model, respectively.Source: Figure 16 · p. 17, Figure 18 · p. 17

D. Multi-Class Targeted Classification Results

E. Bidirectional Reordering Algorithm

  • Algorithm 2 generates 2n−1 visually identical reorderings of a length-n string by forming orderings and encoding them with Unicode controls.The algorithm fragments show recursive swaps and an encoding procedure that concatenates elements into a string.Source: E. Bidirectional Reordering Algorithm · p. 18

F. OCR Defense Algorithm

  • The OCR defense algorithm resolves control characters, renders the input as an image, applies OCR, and returns the recognized text as the model input.Source: F. OCR Defense Algorithm · p. 18
Loading 2106.09898v2…