PaperlayerView exact revision on arXiv

Source-linked AI summary

Imitation Attacks and Defenses for Black-box Machine Translation Systems

Eric Wallace, Mitchell Stern, Dawn Song

arXiv:2004.15015v3cs.CLcs.CRcs.LG

TL;DR

Black-box machine translation systems are valuable targets for theft and attacks, but their internal models are inaccessible. The paper trains imitators from API-labeled monolingual data, transfers adversarial examples from them, and proposes a defense that degrades imitation and attack transfer while costing the defender BLEU and speed.

  • Problem

    Black-box MT systems have commercial value and can make costly errors, motivating investigation of their theft and exploitation.

    Source: §1 · p. 1

  • Method

    The paper trains imitation models from monolingual queries to black-box MT systems, uses them to generate transferable adversarial examples, and modifies translations to misdirect imitation training.

    Source: §1 · pp. 1–2, §2 · p. 2

  • Results

    Bing scores 32.9 BLEU on WMT14 English→German while its imitation scores 32.4, and imitation-model similarity supports attacks producing mistranslations, dropped content, and malicious outputs.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The proposed defense degrades adversary BLEU and adversarial-example transfer, but costs the defender BLEU and inference speed.

    Source: §1 · p. 2

    The defense requires generating and backpropagating 100 translation hypotheses and reduces defender BLEU.

    Source: §5 · p. 9

Abstract

from arXiv · show

Adversaries may look to steal or attack black-box NLP systems, either for financial gain or to exploit model errors. One setting of particular interest is machine translation (MT), where models have high commercial value and errors can be costly. We investigate possible exploits of black-box MT systems and explore a preliminary defense against such threats. We first show that MT systems can be stolen by querying them with monolingual sentences and training models to imitate their outputs. Using simulated experiments, we demonstrate that MT model stealing is possible even when imitation models have different input data or architectures than their target models. Applying these ideas, we train imitation models that reach within 0.6 BLEU of three production MT systems on both high-resource and low-resource language pairs. We then leverage the similarity of our imitation models to transfer adversarial examples to the production systems. We use gradient-based attacks that expose inputs which lead to semantically-incorrect translations, dropped content, and vulgar model outputs. To mitigate these vulnerabilities, we propose a defense that modifies translation outputs in order to misdirect the optimization of imitation models. This defense degrades the adversary's BLEU score and attack success rate at some cost in the defender's BLEU and inference speed.

1 Introduction

The paper investigates stealing and attacking black-box machine translation systems, using imitation models both to reproduce production outputs and to transfer adversarial examples. It also proposes a defense that misdirects imitation training, with costs to the defender.

1 Introduction

  • The paper studies two exploits of black-box MT: training models to imitate production systems, then using those imitators to attack the production systems.The motivation includes avoiding long-term API costs or launching a competitor service, and inducing targeted model errors.Source: §1 · p. 1
  • Imitators are trained on monolingual sentences labeled by the production API, borrowing the student-model approach of knowledge distillation.In the example, English-corpus sentences are labeled through the API before the imitation model is trained.Source: §1 · p. 1, Figure 1 · p. 2
  • Production imitation models come within 0.6 BLEU of three systems across high- and low-resource language pairs, enabling attacks that transfer to production MT.The attacks produce targeted mistranslations, dropped content, or malicious outputs; the proposed defense reduces imitation BLEU and attack transfer at some cost to defender BLEU and inference speed.Source: §1 · pp. 1–2

2 How We Imitate MT Models

The paper treats MT model stealing as imitation from prediction-only access to a black-box victim. It uses monolingual inputs labeled by the victim to train a model intended to match both its held-out accuracy and functional behavior.

2 How We Imitate MT Models

  • The adversary queries a victim’s predictions without probabilities or logits, then trains an imitation model from monolingual inputs and returned translations.The victim’s architecture, hyperparameters, and training data are unknown.Source: §2 · p. 2
  • The imitation goal is comparable held-out accuracy and functional similarity, meaning similar inputs should translate to similar outputs.Functional similarity is sought to enhance adversarial-example transferability.Source: §2 · p. 2
  • Unlike ordinary distillation, stealing typically uses out-of-domain queries and lacks output probabilities needed for distribution-matching losses such as KL divergence.The victim’s training data is unknown, so queried inputs are typically out of domain.Source: §2 · p. 2
  • The paper extends earlier model-stealing work in classification, vision, and other language tasks to machine translation.Prior language-task examples include stolen text classifiers, reading-comprehension models, and textual-entailment models.Source: §2 · p. 2

3 Imitating Black-box MT Systems

The experiments test imitation of local and production MT systems across differences in data and architecture. They report close imitation on in-domain and out-of-domain tests, including production systems and two language pairs.

3.1 Research Questions and Experiments

  • Experiments vary source data and architecture, probing whether imitation remains possible when the victim’s training domain is unknown and its OOD outputs may be incorrect.The paper identifies the effect of out-of-domain data as an open question, unlike architecture mismatch, which prior distillation work had examined.Source: §3.1 · p. 3

3.2 We Closely Imitate Local Models

  • Local imitators match victims on in-domain and OOD test BLEU; their inter-system BLEU reaches 70.5 versus 62.1 for independently seeded Transformers.OOD queries slow imitation but do not prevent it; same-source imitation can learn faster than the victim, likely because model translations are simpler than human translations.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

3.3 We Closely Imitate Production Models

  • Production queries cover high-resource English→German and low-resource Nepali→English; only Google is imitated for Nepali→English because other services do not offer the pair or are low quality.English→German queries use about 4.5 million WMT14 sentences; Nepali queries use Wikipedia and common-crawl data.Source: §3.3 · p. 4
  • English→German production imitations stay within 0.6 BLEU; for Nepali→English, Google scores 22.1 BLEU and its imitation scores 22.0.The English→German table covers production systems and imitation results on WMT newstest2014.Source: Table 2 · p. 4, §3.3 · p. 4
  • English→German imitation remains within 0.9 BLEU on the out-of-domain IWSLT test set, with 65.6 inter-system BLEU on WMT14.Source: §3.3 · p. 4
  • The estimated cost of obtaining the English→German training data is as little as $10.The estimate is specific to the English→German imitation models.Source: §3.3 · p. 4

4 Attacking Production Systems

The paper uses imitation models to attack production MT systems, where direct white-box access is unavailable. It develops gradient-based and universal attacks, then evaluates their transfer to production systems.

4.1 What are Adversarial Examples for MT?

  • Because white-box attacks require access to the target, the paper generates attacks on imitation models and transfers them; it also introduces input-agnostic phrases that cause errors or dropped content.Source: §4.1 · pp. 5–6

4.2 How We Generate Adversarial Examples

  • Attacks iteratively replace input tokens using gradients of an adversarial loss, testing candidate replacements and selecting the one with the lowest loss.The procedure evaluates the top 50 candidates; the authors say using at least 10 is critical for strong results.Source: §4.2 · p. 6

4.3 Types of Adversarial Attacks

  • Four attacks flip a chosen output token, turn altered nonsense into malicious output, append a universal mistranslation trigger, or append a suffix that drops itself and later text.For universal attacks, gradients are averaged across inputs; their losses are set to encourage changed predictions for triggers or minimally changed predictions for suffix droppers.Source: §4.3 · p. 6

4.4 Experimental Setup

  • Flip evaluation tracks attack success and changed-token rate; nonsense tracks prediction-preserving modifications and changed-token rate, while universal triggers use BLEU without a brevity penalty.Source: §4.4 · p. 7
  • Evaluation uses 200 antonym-bearing WMT examples for three attacks and 100 prefix/suffix combinations for the dropper; each universal attack targets one production system.Source: §4.4 · p. 7

4.5 Results: Attacks on Production Systems

  • More than three-quarters of imitation-model attacks succeed; targeted flips usually change about 10% of tokens and transfer to Systran 23%, while Systran’s trigger-output BLEU is 5.46.Appending seven random BPE tokens instead yields BLEU scores of 62.2 and 58.8 for two choices.Source: §4.5 · p. 7
  • Bing drops the appended phrase and following suffix in 76 of 100 universal suffix-dropper inputs.Source: §4.5 · p. 7
  • Attacks from a WMT14-trained Transformer Big transfer to Google 8.8% of the time, about half as often as attacks from the imitation model.The authors report that high-fidelity imitation models considerably enhance transferability.Source: §4.5 · p. 7

5 Defending Against Imitation Models

The defense replaces original translations with high-accuracy alternatives chosen to misdirect imitation-model optimization, trading some victim BLEU and inference speed for lower imitation accuracy and adversarial transfer.

5 Defending Against Imitation Models

  • 34.5→32.7 imitation BLEU versus 34.6→33.8 victim BLEU; inter-system BLEU falls from 69.7 to 63.9, 57.8, and 53.5 at thresholds 90, 80, and 70.The defense outperforms the naïve defense across settings.Source: Figure 3 · p. 8, §5 · p. 9
  • The defense seeks an alternate translation whose imitation-model gradient has low cosine similarity to the gradient from the victim’s original output.Because the adversary’s parameters are unknown, the search seeks high angular deviation across ten Transformer checkpoints.Source: §5 · p. 8
  • Computing the Jacobian is intractable for MT because one output token can have 5,000–50,000 BPE classes, so the authors instead use a search procedure.The defense also requires generating and backpropagating 100 hypotheses, with extra compute and lower BLEU as costs.Source: §5 · pp. 8–9
  • The procedure generates 100 alternate translations from diverse candidate sources, filters them by BLEU Match to the original, then selects the candidate with the greatest gradient deviation.The candidates include beam and diverse-beam outputs, random samples, top-k sampling, and nucleus sampling; lower thresholds allow more steering but can reduce victim accuracy.Source: §5 · p. 8
  • Without defense, malicious-nonsense attacks transfer at 38%; thresholds 90, 80, and 70 reduce transfer to 32.5%, 29.5%, and 27.0%.Using an 11.5% transfer rate between two Transformers as a rough baseline, the defense prevents about 20–40% of the additional errors gained through imitation.Source: §5 · p. 9

6 Conclusion

The conclusion presents model stealing and adversarial examples as practical production-NLP concerns and calls for stronger defenses and greater attention to security and privacy.

6 Conclusion

  • Model stealing and adversarial examples are practical concerns for production NLP systems, not merely hypothetical risks.The authors cite Bing’s copying of Google search outputs as an example of model stealing.Source: §6 · p. 9
  • A naïve defense that equally degrades victim and imitation BLEU could amount to deploying a worse MT system.Source: §6 · p. 9
  • The authors hope to improve and deploy NLP defenses and make security and privacy a more prominent research focus.Source: §6 · p. 9

Addressing Potential Ethical Concerns

The authors describe steps to reduce real-world harm and support long-term benefit, and report their research framework and model configurations.

Addressing Potential Ethical Concerns

  • The authors avoided harm to real users, used deliberately ludicrous attacks, deleted imitation data and models, and reported vulnerabilities and their defense to Google, Bing, and Systran.They acknowledge possible short-term reputational harm while arguing the work could help improve MT over the long term.Source: Addressing Potential Ethical Concerns · p. 9

A Framework and Hyperparameters

  • Experiments used fairseq and TPU v3-8 devices; IWSLT used dataset-associated architectures, while production-model imitation used Transformer Big.The reported BPE setup used SentencePiece, with vocabulary sizes of 10,000, 32,768, and 10,000 tokens.Source: A Framework and Hyperparameters · p. 12

B Example Translations

Table 6 presents example translations for production MT systems and their imitation models.

B Example Translations

C Estimated Data Collection Costs

The section estimates the cost of collecting WMT14 English→German data through official translation APIs or by scraping a public demo. It compares API charges with the compute cost of automated scraping.

C Estimated Data Collection Costs

  • The cost estimates compare per-character charges for official APIs with scraping an online translation demo using automated cloud machines.The authors consider both public-facing data-acquisition routes.Source: C Estimated Data Collection Costs · p. 12
  • For the approximately 640,654,771-character English side of WMT14 English→German, the displayed cost calculation is $12,803; Bing’s $6,000 USD subscription covers up to one billion characters per month.8The calculation is (640,654,771 - 500,000) × $20 / 1,000,000; Bing’s subscription leaves 359,345,229 characters unused.Source: C Estimated Data Collection Costs · p. 12
  • Roughly $10 in EC2 machine time would scrape 4,468,840 WMT14 source sentences at five seconds per query, totaling 6,207 machine hours.The process could be accelerated by parallelizing queries across cloud machines.Source: C Estimated Data Collection Costs · p. 12

D Universal Suffix Dropper Evaluation

The Universal Suffix Dropper evaluation pairs benign-looking prefixes with adversarial-message suffixes, while the related-work discussion surveys defenses against model stealing and adversarial examples.

D Universal Suffix Dropper Evaluation

  • The Universal Suffix Dropper is evaluated on the Cartesian product of ten benign-looking prefixes and ten suffixes resembling secret messages.The prefixes include encyclopedic statements about topics such as email and Earth; suffixes include purported secret codes and messages.Source: D Universal Suffix Dropper Evaluation · pp. 12–13
  • A WMT14 English→German validation example shows imitation models producing highly similar outputs to the official APIs.The comparison is reported for official API outputs and imitation models as of December 2019.Source: Table 6 · p. 13

E Existing Adversarial Defenses

  • Rate limiting and output randomization can slow model stealing but do not prohibit it; query monitoring may detect stealing, though out-of-distribution detection can flag benign users.The cited monitoring approach assumes consecutive out-of-distribution queries.Source: E Existing Adversarial Defenses · p. 13
  • Watermarking tests whether a suspected stolen model reproduces intentionally incorrect responses, but fine-tuning can subvert these defenses.The discussion cites Zhang et al. (2018), Szyller et al. (2019), Krishna et al. (2020), Hisamoto et al. (2020), and Chen et al. (2019).Source: E Existing Adversarial Defenses · p. 13
  • Adversarial training can empirically improve MT robustness, while cited provable word-replacement defenses apply only to shallow neural classification models.The cited work includes Goodfellow et al. (2015), Ebrahimi et al. (2018), Jia et al. (2019), and Huang et al. (2019).Source: E Existing Adversarial Defenses · p. 13
  • A language model can provide empirical robustness against the malicious nonsense attack by detecting ungrammatical source inputs.Source: E Existing Adversarial Defenses · p. 14

F Angular Deviations Of Defense

Figure 4 reports how often the defense returns the original output at three BLEU thresholds and plots angular deviations for other cases.

F Angular Deviations Of Defense

  • Figure 4’s histogram plots angular deviation, defined as the arccosine of the cosine similarity between g and ˜g.Source: F Angular Deviations Of Defense · p. 14, Figure 4 · p. 14
  • 71.1%, 62.3%, and 72.84% are the rates of original-output returns at 70, 80, and 90 BLEU thresholds, respectively.Source: Figure 4 · p. 14
  • The defense returns the original output when no candidate meets the BLEU threshold or when angular deviation is low.Source: Figure 4 · p. 14

G Adversarial Attack Screenshots

Figures 5–11 are screenshots of the attacks working on production systems as of April 2020.

G Adversarial Attack Screenshots

Loading 2004.15015v3…