PaperlayerView exact revision on arXiv

Source-linked AI summary

Fall of Giants: How popular text-based MLaaS fall against a simple evasion attack

Luca Pajola, Mauro Conti

arXiv:2104.05996v1cs.CRcs.LG

TL;DR

Text-domain adversarial attacks have mainly targeted model weaknesses, while the indexing stage of text pipelines has received less attention. The paper proposes ZeW, which injects invisible Unicode characters to affect indexing, and finds vulnerabilities in most of the commercial services it tests. It also presents an input-validation countermeasure and identifies conditions that limit attack efficacy.

  • Problem

    Text-domain attacks have mainly targeted model weaknesses, while possible vulnerabilities in other pipeline stages, including indexing, have received less attention.

    Source: §1 · p. 2

  • Method

    ZeW injects invisible zero-width Unicode characters into text to affect indexing, and the paper evaluates it on hateful sentences and commercial MLaaS.

    Source: §1 · p. 2

  • Results

    11 of 12 tested services showed vulnerabilities, with 8 heavily affected by ZeW.

    Source: §9 · p. 12

  • Takeaways & Limitations

    The authors argue that simple input validation should be integrated into real-life NLP tools to flag unlikely characters used by ZeW.

    Source: §8 · p. 12

    ZeW's efficacy depends on service implementation: character-based models are more resilient, and discarding unrecognized characters makes the attack unsuccessful.

    Source: §8 · p. 11

Abstract

from arXiv · show

The increased demand for machine learning applications made companies offer Machine-Learning-as-a-Service (MLaaS). In MLaaS (a market estimated 8000M USD by 2025), users pay for well-performing ML models without dealing with the complicated training procedure. Among MLaaS, text-based applications are the most popular ones (e.g., language translators). Given this popularity, MLaaS must provide resiliency to adversarial manipulations. For example, a wrong translation might lead to a misunderstanding between two parties. In the text domain, state-of-the-art attacks mainly focus on strategies that leverage ML models' weaknesses. Unfortunately, not much attention has been given to the other pipeline' stages, such as the indexing stage (i.e., when a sentence is converted from a textual to a numerical representation) that, if manipulated, can significantly affect the final performance of the application. In this paper, we propose a novel text evasion technique called "\textit{Zero-Width} attack" (ZeW) that leverages the injection of human non-readable characters, affecting indexing stage mechanisms. We demonstrate that our simple yet effective attack deceives MLaaS of "giants" such as Amazon, Google, IBM, and Microsoft. Our case study, based on the manipulation of hateful tweets, shows that out of 12 analyzed services, only one is resistant to our injection strategy. We finally introduce and test a simple \textit{input validation} defense that can prevent our proposed attack.

1. Introduction

MLaaS makes capable models accessible without requiring users to train them, while text attacks have largely targeted model weaknesses and readability-preserving perturbations. This paper introduces an attack on text indexing and evaluates it on commercial services.

1. Introduction

  • MLaaS lets organizations use capable models without managing complex training, addressing barriers including data, computing resources, and ML expertise.The market was valued at 1.0 billion USD in 2019, with an estimate of 8.48 billion USD by 2025.Source: §1 · p. 1
  • ZeW injects invisible zero-width Unicode characters to disrupt text indexing, a pipeline stage that converts text into numerical representations.The attack targets a pipeline stage rather than the ML model itself.Source: §1 · p. 2
  • The authors study ZeW’s effects on indexing strategies and commercial services through a case study manipulating hateful sentences.The injection strategy inserts one character within a negative word or one between each of its characters.Source: §1 · p. 2
  • 11 of 12 tested services showed vulnerabilities, with 8 strongly affected by ZeW.The tested MLaaS came from Amazon, Google, IBM, and Microsoft.Source: §1 · p. 2
  • The paper proposes a countermeasure to ZeW that can be integrated into text ML pipelines.The introduction describes the countermeasure as simple.Source: §1 · p. 2

2. Background and Preliminaries

The background introduces text-processing pipelines, application security beyond the model, and challenges that distinguish adversarial attacks in text from those in computer vision.

2.1. Text Pipeline

  • Indexing converts text into numerical vectors using word-count or TF-IDF, one-hot, or dense representations such as embeddings.A vocabulary is defined at training time for word- or character-level representations.Source: §2.1 · p. 3
  • Unrecognized words or characters may be discarded or mapped to UNK, which can collapse distinct rare names into the same representation.The text pipeline also includes original documents, preprocessing, indexing, and an ML model.Source: §2.1 · pp. 2–3

2.2. Adversarial Machine Learning & Application Security

  • Application security must cover preprocessing and transformations as well as the ML algorithm; ZeW targets text preprocessing and indexing engines.Source: §2.2 · p. 4

2.3. Challenges of adversaries in Text-Domain

  • Text is discrete, small changes can be visible to people, and character or word edits can change sentence meaning, complicating readable adversarial perturbations.Text attacks therefore face different input, perception, and semantic challenges from image attacks.Source: §2.3 · p. 4

3. Zero-Width Attack

The section introduces ZeW, which uses invisible Unicode characters to disrupt text indexing while preserving human readability. It explains how indexing choices shape the attack and proposes input validation against it.

3.1. Motivations

  • ZeW exploits invisible Unicode characters to insert arbitrary perturbations without changing a sentence’s human-readable meaning, addressing a gap in attention to indexing-stage security.The authors say they are the first, to their knowledge, to examine zero-width characters in adversarial machine learning.Source: §3.1 · pp. 4–5

3.2. Theoretical Perspective

  • For word-based representations, poisoned words may become UNK tokens or be discarded, changing the representation or removing information from the sentence.If enough words are discarded, the resulting sentence could be empty.Source: §3.2 · p. 5
  • For character-based representations, mapping injected characters to UNK adds noise, while discarding them leaves the poisoned sentence equivalent to the original.In the discard case, the attack has no effect.Source: §3.2 · p. 5
  • ZeW generally increases noise or reduces information in sentence representations, and the paper also outlines poisoning and trojan scenarios while leaving their definitions out of scope.The poisoning scenario assumes access to training data; the trojan scenario uses a rare zero-width sequence as a test-time trigger.Source: §3.2 · p. 5

3.3. Countermeasure

  • A filter rejecting non-printable characters or a sanitizer discarding them can prevent ZeW at the input-validation stage.The authors place these controls before model preprocessing and inference.Source: §3.3 · p. 5

4. Case Study: Hate Speech Manipulation

The case study tests ZeW against text-based services from Amazon, Google, IBM, and Microsoft using hate-speech manipulation. Its injector targets individually negative words with one of two zero-width-character masks.

4.1. Overview

  • The study tests popular text MLaaS, including sentiment analyzers and translators, to assess whether ZeW affects their outcomes and the magnitude of vulnerability.Hate-speech manipulation is used as the case study; an attack counts as successful if injection affects target-model performance.Source: §4.1 · p. 6

4.2. Manipulation algorithm

  • The injector identifies words with negative polarity scores and inserts zero-width characters into those words to manipulate hateful sentences.The approach assumes hateful sentences contain a negative part of speech.Source: §4.2 · p. 6
  • Mask1 inserts one random zero-width space inside a target word, while Mask2 inserts multiple random characters between its letters.The two masks provide different injection levels; VaderSentiment identifies negative words.Source: §4.2 · p. 6

5. Results on Controlled Environments

Controlled experiments test ZeW on sentiment classifiers trained with different text representations. The reported attack success and model comparisons vary with tokenization and handling of unknown tokens.

5.1. Experimental Settings

  • The experiments use Sentiment140’s 1.6 million labeled tweets, split into 70% training, 10% validation, and 20% testing, with SGD and RNN classifiers across character- and word-based representations.The models use standard preprocessing that removes hashtags, mentions, and URLs.Source: §5.1 · p. 6

5.2. Results and Considerations

  • DNN-based models tend to outperform linear models in accuracy, a gap the authors link to the TfidfVectorizer’s memory-limited vocabulary; unknown tokens did not boost performance.The evaluation compares six models.Source: §5.2 · p. 7
  • ASR was always under 40%; the authors attribute this partly to sentence polarity depending on token sequences and to ZeW’s insertion-only operations.The attack modifies individually negative tokens, while a sentence’s polarity may arise from a sequence of tokens.Source: §5.2 · p. 7
  • ZeW affects models using different tokenization strategies; character-based models are generally more resilient, and discarding unknown tokens makes the attack fail.Models that consider unknown tokens are generally more vulnerable.Source: §5.2 · p. 7

6. Results on MLaaS

The MLaaS evaluations show that ZeW’s effects vary by service and task, from limited changes in toxicity detection to pronounced changes in sentiment, insight extraction, and translation. The experiments also assess sanitization as a countermeasure and discuss limits on interpreting the observed patterns.

6. Results on MLaaS

  • The MLaaS evaluation groups services into hate-speech detection, insight extraction, sentiment analysis, and translation, and does not compare ZeW directly with state-of-the-art attacks.The authors distinguish their focus on disrupting indexing from attacks targeting ML algorithms, while noting that their attack model allows unrestricted modifications.Source: §6 · p. 7

6.1. Dataset & Evaluation on VaderSentiment

  • VaderSentiment’s median negativity falls from 0.35 on Real to 0.0 for both masks, and sanitization restores the original distribution.The evaluation uses 1,094 hateful sentences in each masked corpus after discarding Real sentences without a detected negative word; Table 1 describes model ACC and attack success percentage (ASP).Source: §6.1 · p. 7, Table 1 · p. 8

6.2. Hate Speech Detection

  • Toxicity detectors are comparatively resistant: Google Perspective’s median confidence shifts from 0.95 on Real to 0.84 on Mask1 and 0.83 on Mask2.Microsoft Moderator shifts from 0.99 on Real to 0.97 on Mask1 and 0.86 on Mask2; the authors nevertheless conclude both services are vulnerable.Source: §6.2 · p. 8, Figure 4 · p. 9

6.3. Insights Extractors

  • IBM Watson Tone Analyzer’s median Jaccard similarity is 0.5 for Mask1 and 0.33 for Mask2; Watson Personal Insight also changes across the masked corpora.Forty percent of Tone Analyzer sentences are unaffected; 322 sentences without a detected tone are excluded. Personal Insight differs on two dimensions for Mask1, while Mask2 moves all five dimensions close to zero.Source: §6.3 · p. 9, Figure 5 · p. 9, Figure 6 · p. 9

6.4. Sentiment Analyzers

  • Across sentiment analysis, three of four services are severely vulnerable, while Amazon Comprehend resists both injection modalities.Google Cloud Natural Language’s median moves from 0.5 to 0.2; IBM and Microsoft medians also decrease under both masks.Source: §6.4 · p. 10

6.5. Translators

  • All four translation services are vulnerable: Amazon and IBM resist Mask1 but not Mask2, while Google and Microsoft are vulnerable to both.The authors report greater difficulty handling Mask2 across all models and describe Amazon as weakly vulnerable overall.Source: §6.5 · p. 10

6.6. Considerations

  • Across the tested services, the authors identify differing vulnerability patterns and leave the discrepancy in translators’ Mask1 resistance unexplained.They suggest that Mask2’s stronger impact may indicate character-based models, but call for more investigation of neural machine-translator architectures.Source: §6.6 · p. 10

7. Related Work

The section surveys adversarial attacks against the commercial MLaaS considered in the paper, spanning hate-speech detectors, sentiment analyzers, and machine translators.

Hate Speech Detectors

  • Figure 7 depicts negative-score distributions and medians; it defines a sentiment service as vulnerable when attacked and Real distributions differ.The blue line marks the median.Source: Figure 7 · p. 11
  • Prior hate-speech detector attacks used sentence mistakes and acoustic or visual similarities, with reported evasion powers of 33% and 72.5% for the latter techniques,,.The reported percentages belong to the acoustic- and visual-similarity techniques, respectively.Source: §7 · pp. 10–11

Sentiment Analyzers

  • Sentiment attacks include Fast Gradient Sign Attack, DeepFool, semantic-similarity word replacement, and the black-box TextBugger framework.DeepFool uses Word Mover’s Distance to find influential words.Source: §7 · p. 11

Machine Translators

  • Machine-translation attacks include gradient-based Adv-Gen and Seq2Seq methods targeting non-overlapping outputs or keyword inclusion.The non-overlapping attack seeks novel adversarial sentences; the keyword attack seeks translations containing target keywords.Source: §7 · p. 11

8. Limitations

ZeW’s effectiveness depends on service implementation, and the proposed sanitization is a ZeW-specific patch rather than a general defense.

Attack

  • ZeW’s efficacy depends on implementation: character-based models are more resilient than word-based ones, and discarding unrecognized characters makes the attack unsuccessful.The attack also offers limited control: it can affect a translation without steering its output, or reduce a class likelihood without forcing a target class.Source: §8 · p. 11

Defense

  • The proposed sanitization removes zero-width characters, and comparing results for original and sanitized sentences can reveal discrepancies.The approach is possible because normal English sentences should not contain these characters.Source: §8 · p. 11

Attack

  • Translator-service vulnerability is assessed by whether attacked-corpus BLEU-score distributions are not close to one.The caption says the violin plots show BLEU distributions and blue median lines.Source: Figure 8 · p. 12

Defense

  • Sanitization applies only to ZeW, making it a patch rather than a general solution; limiting character vocabulary is proposed as a broader countermeasure.The paper also notes that character-based models show intrinsic resilience to ZeW.Source: §8 · p. 12

9. Conclusions

The paper proposes ZeW, an injection attack that can disrupt indexing while keeping text readable, and finds vulnerabilities in 11 of 12 commercial services. It concludes that security analysis should cover the full ML pipeline, not only the model.

9. Conclusions

  • ZeW injects non-printable Unicode characters into malicious sentences, potentially disrupting indexing while preserving full readability.The paper frames this as an alternative to the readability constraint described as a major obstacle in text adversarial ML.Source: §9 · p. 12
  • 11 of 12 services were vulnerable; among them, 3 showed good resistance and 8 were heavily affected.The services were developed by Amazon, Google, IBM, and Microsoft.Source: §9 · p. 12
  • The paper proposes input validation as a countermeasure and describes potential use beyond hate-speech manipulation, including obfuscating webpages analyzed for counter-terrorism.The countermeasure is recommended for real-life NLP tools in that scenario.Source: §9 · p. 12
  • The authors argue that security research should examine pipeline components beyond the ML model, including vulnerabilities arising from multiple text representations such as ASCII and Unicode.The paper notes that security challenges differ across computer vision, natural language processing, and signal domains.Source: §9 · pp. 12–13
Loading 2104.05996v1…