PaperlayerView exact revision on arXiv

Source-linked AI summary

A Reinforced Generation of Adversarial Examples for Neural Machine Translation

Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, Jiajun Chen

arXiv:1911.03677v2cs.CLcs.LG

TL;DR

NMT systems can fail on less-decent inputs, and it remains difficult to understand when and how they fail. This paper uses reinforcement learning and a discriminator to generate meaning-preserving adversarial examples for translation. The authors report stable degradation across victim models and describe the attacks as efficient and useful for exposing pitfalls.

  • Problem

    Understanding when and how NMT systems fail on less-decent inputs is difficult, while handcrafted error features are limited and costly to develop.

    Source: §1 · pp. 1–2

  • Method

    The paper uses reinforcement learning with a discriminator as a terminal signal to learn discrete source-token perturbations that target translation-metric degradation.

    Source: §1 · p. 2

  • Results

    The method achieves stable degradation with meaning-preserving adversarial examples across different victim models.

    Source: §4.4 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    The authors report that monolingual-data generation makes mass production of examples for victim-model analysis and robustness improvement convenient.

    Source: §7 · p. 9

Abstract

from arXiv · show

Neural machine translation systems tend to fail on less decent inputs despite its significant efficacy, which may significantly harm the credibility of this systems-fathoming how and when neural-based systems fail in such cases is critical for industrial maintenance. Instead of collecting and analyzing bad cases using limited handcrafted error features, here we investigate this issue by generating adversarial examples via a new paradigm based on reinforcement learning. Our paradigm could expose pitfalls for a given performance metric, e.g., BLEU, and could target any given neural machine translation architecture. We conduct experiments of adversarial attacks on two mainstream neural machine translation architectures, RNN-search, and Transformer. The results show that our method efficiently produces stable attacks with meaning-preserving adversarial examples. We also present a qualitative and quantitative analysis for the preference pattern of the attack, demonstrating its capability of pitfall exposure.

1 Introduction

NMT can fail on real-world inputs, while understanding when and how it fails is difficult; handcrafted error features are costly and may not generalize across architectures. The paper proposes reinforcement-learning-based adversarial generation with a discriminator to constrain semantics and expose translation-system pitfalls.

1 Introduction

  • NMT systems can fail on less-decent real-world inputs, and determining when and how they fail remains challenging.A typo omitting the Chinese character “炸” can cause significant English translation errors, although “爆” and “爆炸” both mean “bombing.”Source: §1 · p. 1, Table 1 · p. 1
  • Handcrafted error features require linguistic and architectural expertise and may be costly or inapplicable when errors are hard to formulate or architecture-specific.Prior adversarial approaches also face discrete-token generation and semantic-preservation challenges.Source: §1 · pp. 1–2
  • The proposed RL paradigm uses a discriminator as a terminal signal to constrain semantics while learning discrete token perturbations aimed at degrading translation metrics.Experiments are reported to produce semantically constrained adversarial examples and effective attacks.Source: §1 · p. 2
  • The paper presents its method as efficient, model-agnostic, and usable across machine-translation architectures without handcrafted error features.The stated input requirement is source data alone.Source: §1 · p. 2
  • The authors analyze attacks on Transformer to assess whether attack preferences can expose system pitfalls.The introduction presents this as a qualitative and quantitative analysis of the method’s pitfall-exposure capability.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe NMT as conditional, autoregressive translation and introduce actor-critic reinforcement learning. They also frame NLP adversarial generation as constrained perturbation, often pursued through gradient-based search.

2.1 Neural Machine Translation

  • NMT encodes the source into hidden representations, then uses an attention-equipped decoder to generate each target token autoregressively until EOS.The decoder’s attention context is computed for its current state from the encoder representations.Source: §2.1 · p. 2

2.2 Actor-Critic for Reinforcement Learning

  • Actor-critic agents use a policy network, the actor, to choose actions and a value network, the critic, to estimate state values within a reward-maximizing process.The overview figure depicts the paper’s agent receiving environment states and being updated from accumulated rewards.Source: §2.2 · pp. 2–3, Figure 1 · p. 3

2.3 adversarial examples in NLP

  • Adversarial-example generation seeks an input perturbation that degrades system performance subject to a constraint, while NLP methods commonly use gradient-based surrogate-loss search.Because the search space is large, the described approach perturbs a small ratio of token positions and greedily searches candidates.Source: §2.3 · p. 3

3 Approach

The approach treats the victim translation model as part of an environment that rewards translation degradation while a discriminator screens out ill-perturbed inputs. A sequential agent learns token-level perturbations and applies them at inference using its critic’s value estimates.

3 Approach

  • The agent modifies source tokens sequentially, while the environment returns rewards for translation degradation and a discriminator supplies per-step survival signals.The victim translation model is part of the environment; the discriminator terminates modifications judged ill-perturbed.Source: §3 · pp. 3–4, §3.1 · p. 4, §3.1.2 · p. 4

3.1.1 Environment State

  • The environment state combines the source sequences with the current token position, then updates after each agent modification until termination or the end of the sequence.Source: §3.1.1 · p. 4

3.1.2 Reward Process with Discriminator

  • During training, the agent receives survival rewards at each step and a final metric-degradation reward if it survives to the end.The discriminator judges whether a modified source still matches the original target; the final reward uses the difference between original and perturbed output scores.Source: §3.1.2 · p. 4
  • The discriminator uses bidirectional GRU encoders for source and target, and its parameters are updated alongside the agent’s using perturbed negative samples.Source: §3.1.2 · p. 4

3.2 Agent

  • The actor chooses whether to perturb each token and the critic estimates state value; at inference, the agent selects the nearest candidate only when the critic’s value is positive.During training, the agent samples perturbation decisions and candidates stochastically; candidates are selected from nearby victim-vocabulary embeddings, with UNK assigned when none is nearby.Source: §3.2 · p. 5

4 Experiments

Across the reported translation tasks and models, the method produces attacks with high human-evaluated semantic similarity and comparable relative degradation to gradient search. The experiments also examine model settings, evaluation metrics, and the kinds of edits the attacks prefer.

4.1 Data Sets; 4.2 Victim Models

  • Attacks are evaluated on Zh→En, En→Fr, and En→De using RNN-search and Transformer victim models, with word- and subword-level Transformer settings for Zh→En.Source: §4.1 · p. 5, §4.2 · p. 5

4.1 Data Sets

  • The agent is trained using the parallel data used for the victim models, and adversarial examples are generated from NIST test sets for Zh→En and WMT test sets for En→De and En→Fr.Source: §4.1 · p. 5

4.3 Metrics

  • The evaluation reports source modification rate via chrBLEU and target translation degradation via relative decrease in BLEU, alongside human evaluation of source semantic similarity.Human evaluation uses a 0–5 scale and a double-blind assessment of a random 10% sample mixed with baselines.Source: §4.3 · pp. 5–6

4.4 Results

  • HE remains above 3.7 across models and language pairs, while the method achieves relative degradation comparable to gradient search without handcrafted semantic constraints.The method uses the same training setting across the reported models and language pairs.Source: Table 3 · p. 6, Table 4 · p. 6, §4.4 · p. 6

4.5 Case Study

  • The case study finds that the method is less likely than search to change numbers into different values, favoring edits to other forms instead.The authors describe its resulting analyses as more insightful and plausible.Source: §4.5 · p. 6

5 Analysis

The analysis compares generation efficiency and attack patterns, tests cross-architecture attacks and adversarial tuning, and examines perturbed examples that can improve translation. It reports faster generation than search under the stated memory limit and describes distinct attack preferences across parts of speech.

5.1 Efficiency

  • Table 5 contrasts the agent’s meaning-preserving perturbations with search examples that can make substantial semantic changes.Its examples include number and quantifier changes and a suffix change.Source: Table 5 · p. 7
  • Under a 2.5G memory limit, the method generates examples significantly more efficiently than the search paradigm for the same 800 Zh→En inputs.Figure 2 compares time consumption on a single Nvidia 1080.Source: §5.1 · p. 6, Figure 2 · p. 7

5.2 Attack Patterns

  • At similar modification rates, the agent favors some POS tags, including Ni, Nh, Nl, M, and K, while RSNI is nearly even across tags.The authors associate Ni, Nh, Nl, and M with hard-to-translate parts, and report that the agent favors K, which is less noticed.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3 Attack Generalization

  • Table 6 presents attacks on architectures different from the one used to train the agent.Its caption identifies the trained architecture for each agent, but the supplied passage does not report the table’s outcomes.Source: Table 6 · p. 8

5.4 Tuning with Adversarial Examples

  • Adversarial tuning improves noisy-test BLEU by +4.83 while reducing clean-test BLEU by 0.46 on Zh→En Transformer-BPE.The authors also report improvement on IWSLT test data; Table 7 describes tuning with adversarial examples.Source: §5.4 · p. 8, Table 7 · p. 8

5.5 Reinforced Examples for Machine translation

  • The authors find minor perturbed examples that improve translation on some samples, but not overall, and show two examples for Zh→En Transformer-BPE.The examples alter punctuation and a word with a similar meaning; the authors say such noisy inputs can be applied at test time without model updates.Source: §5.5 · pp. 8–9, Table 8 · p. 9

6 Related Work

Prior NMT adversarial-example methods use embedding or sequence-representation perturbations, gradient-guided token search, and other generation paradigms. The section contrasts their constraints and applicability, including semantic control, gradient cost, and tokenization sensitivity.

Representation perturbation

  • Embedding- and sequence-representation perturbation methods were described as applicable to simple NLP classification but inadequate for MT’s higher semantic constraints; adding accessory annotations further limits applicability.The cited methods include Cheng et al. (2018a, 2018b) and Zhao et al. (2017).Source: §6 · p. 9

Search-based methods

  • Gradient-surrogate search methods face high computation costs, and fixed-tokenization losses can be invalidated when an attack changes tokenization.The section notes that Ebrahimi et al. (2018) focused on character-level systems and Michel et al. (2019) excluded tokenization-changing attacks.Source: §6 · p. 9

Other generation paradigms

7 Conclusion

The paper concludes that its reinforcement-learning paradigm generates meaning-preserving adversarial examples for NMT and exposes translation pitfalls without handcrafted error features. It also highlights efficient generation from monolingual data and mentions reinforced samples as future work.

Conclusion

  • The proposed paradigm exposes NMT translation pitfalls without handcrafted error features and achieves stable degradation with meaning-preserving examples across victim models.The conclusion describes generation from monolingual data as efficient and notes that mass production can support victim-model analysis and robustness improvement.Source: §7 · p. 9
  • Exceptional cases called “reinforced samples” are identified as future work.The conclusion also states that such cases were observed alongside adversarial examples.Source: §7 · p. 9

A Training Details for Agent

The agent and discriminator training setup uses BLEU as a score, batch-based training, and specified optimization and stopping settings. The agent takes around 30 hours to converge on one Nvidia 1080ti, with stronger semantic constraints increasing training time.

Training setup

  • Training uses BLEU as the score and batches of 50 sequence pairs to stabilize training and support early exploration.The stated discriminator accuracy bound is 0.85; the passage gives a and b in Eq. 8 as 0.5 and 10.Source: A Training Details for Agent · p. 11

Optimization

  • Asynchronous reinforcement learning uses a discount factor of 0.99, Adafactor, a 0.001 initial learning rate with 100 warmup steps, and a candidate set of 12.The passage also gives α and β in Eq. 5 as 0.5 and 0.05.Source: A Training Details for Agent · p. 11

Convergence

  • The agent takes around 30 hours to converge on one Nvidia 1080ti, and stronger semantic constraints increase training time.The passage links higher discriminator accuracy bounds and lower convergence boundaries to stronger constraints.Source: A Training Details for Agent · p. 11

B Search-based Attack

The search-based attack follows prior search strategies and uses a reinforced-training algorithm with an agent and discriminator. The algorithm initializes an environment state, samples actions, and uses discriminator accuracy in its stopping procedure.

B Search-based Attack

  • The approach follows search-based adversarial-generation strategies associated with Ebrahimi et al. (2018) and Michel et al. (2019).The passage describes search-based adversarial generation as widely applied in robustness work on machine translation systems.Source: B Search-based Attack · p. 11
  • The discriminator is trained using the initial agent and a batch size set to three times the average convergence episodes.The passage gives this as an implementation setting.Source: B Search-based Attack · p. 11
  • The reinforced-training algorithm alternates discriminator updates with agent interaction, initializing an environment state before sampling actions.The procedure checks discriminator accuracy against a bound and tests that accuracy for stopping.Source: B Search-based Attack · p. 11
  • The agent performs actions and receives rewards, with entropy computed from its output and gradients computed for the loss.The passages also distinguish terminal and non-terminal states, associating V(s_t) with non-terminal states.Source: B Search-based Attack · p. 11

30 end

The adversarial loss is specified for both RNN-search and Transformer, with its token probabilities calculated using the corresponding target. The search procedure samples positions and greedily searches candidate substitutions.

30 end

  • The adversarial loss is applicable to both RNN-search and Transformer, and its token probabilities are calculated using Eq.1 for the corresponding target.The passage introduces the derivation of Ladv in Eq.7.Source: §30 · p. 11
  • Search samples a small ratio of positions and greedily tests candidates using the corresponding loss, with the model's candidate set replacing naive KNN candidates.Both the baseline and proposed model share UNK generation; Chinese uses homophone replacement, while English uses the strategy by Michel et al. (2019).Source: §30 · p. 12
  • Greedy search is used as an approximation, which Ebrahimi et al. (2018) suggest is good enough.The passage presents this as the rationale for greedy search.Source: §30 · p. 12
Loading 1911.03677v2…