PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Neural Machine Translation with Doubly Adversarial Inputs

Yong Cheng, Lu Jiang, Wolfgang Macherey

arXiv:1906.02443v1cs.CL

TL;DR

NMT models can be brittle to small input perturbations, creating a need for robust translation methods. The paper uses gradient-based adversarial source and target inputs to attack and defend the model, achieving improvements over Transformer on Chinese-English and English-German benchmarks and higher robustness on noisy data.

  • Problem

    NMT models remain vulnerable to small input perturbations that can produce different and often incorrect translations.

  • Method

    AdvGen uses translation-loss-guided gradients to generate adversarial source inputs for attack and adversarial target inputs for defense.

  • Results

    2.8 and 1.6 BLEU points are gained over Transformer on Chinese-English and English-German translation benchmarks, with higher robustness on noisy data.

  • Takeaways & Limitations

    The approach improves both translation performance and robustness in the evaluated Chinese-English and English-German tasks.

Abstract

from arXiv · show

Neural machine translation (NMT) often suffers from the vulnerability to noisy perturbations in the input. We propose an approach to improving the robustness of NMT models, which consists of two parts: (1) attack the translation model with adversarial source examples; (2) defend the translation model with adversarial target inputs to improve its robustness against the adversarial source inputs.For the generation of adversarial inputs, we propose a gradient-based method to craft adversarial examples informed by the translation loss over the clean inputs.Experimental results on Chinese-English and English-German translation tasks demonstrate that our approach achieves significant improvements ($2.8$ and $1.6$ BLEU points) over Transformer on standard clean benchmarks as well as exhibiting higher robustness on noisy data.

1 Introduction

NMT can be highly vulnerable to small input perturbations, motivating a white-box, gradient-based approach that jointly attacks and defends translation models with adversarial inputs. On Chinese-English and English-German benchmarks, the approach improves over Transformer while also targeting robustness to noisy data.

  • Motivation: Small input perturbations can produce different and often incorrect translations, even when they do not alter sentence meaning.The paper distinguishes natural annotation noise and artificial deviations generated by attack models as sources of perturbed examples.
  • Approach: AdvGen constructs adversarial examples with gradients guided by the final translation loss from clean NMT inputs.The method addresses the discrete nature of words and the difficulty of perturbing sequence-generation models.
  • Approach: Doubly adversarial training attacks through adversarial source inputs and defends through adversarial target inputs in the decoder.Encoder perturbations target model errors, while decoder perturbations help defend prediction errors.
  • Results: The approach is presented as improving both translation performance and robustness across two common translation benchmarks.The contribution claims significant improvements over the previous state-of-the-art Transformer model.
  • Results: 2.8 and 1.6 BLEU points are gained over state-of-the-art models including Transformer on Chinese-English and English-German benchmarks.The reported improvements are on standard clean benchmark datasets; further experiments assess noisy text robustness.

2 Background

NMT uses an encoder-decoder architecture to model translation probabilities over parallel data. The background introduces adversarial examples as minimally perceptible perturbations intended to distort model outputs, while noting that their imperceptibility is difficult to define for language.

  • Neural Machine Translation: NMT typically uses an encoder-decoder architecture trained to maximize the likelihood of a parallel corpus.The encoder processes source sentences and the decoder generates target sentences conditioned on encoded representations and target inputs.
  • Neural Machine Translation: The encoder maps source words to embeddings and continuous hidden representations, while the decoder maps target inputs to generated outputs.In the standard model, the decoder input is a shifted copy of the target output beginning with a start symbol.
  • Adversarial Examples: An adversarial example corrupts an original input with a small perturbation that remains less perceptible but dramatically distorts model output.White-box examples use information from the attacked model, whereas black-box methods lack access and often rely on prior knowledge.
  • Adversarial Examples: Adversarial-example generation balances the likelihood that a sample is adversarial against the imperceptibility of its perturbation.The formulation uses J(·) for adversariality and R(x′, x) for perturbation imperceptibility; classification commonly bounds perturbations with the l∞ norm.

3 Approach

The approach uses gradient-based AdvGen to create model-specific adversarial source and target inputs, jointly attacking and defending NMT during end-to-end training. Source perturbations attack translation predictions, while target perturbations focus defense on errors associated with perturbed source words.

  • Adversarial Input Generation: AdvGen generates adversarial examples using translation-loss gradients while constraining replacements to preserve semantic similarity.Exact optimization is approximated greedily because solving the objective is intractable.
  • Attack with Adversarial Source Inputs: Source adversarial inputs are generated by sampling replacement positions uniformly and selecting candidates from a small, language-model-based dynamic vocabulary.The dynamic candidate set is much smaller than the full source vocabulary, making generation computationally feasible.
  • Attack with Adversarial Source Inputs: The resulting source example x′ is used with the original target y as training data to improve robustness against source perturbations.Source perturbations can introduce decoder errors that accumulate and substantially change predictions.
  • Defense with Adversarial Target Inputs: Target adversarial inputs z′ are generated to defend against source-induced decoder errors, with target positions selected according to attention from perturbed source words.Unlike source positions, target positions are not sampled uniformly; the attention matrix identifies relevant target words.
  • Training: The final training objective combines clean, language-model, and robustness losses, while AdvGen is treated as a data generator without backpropagating through it.The implementation incurs at most a 20% time overhead compared with the standard Transformer model.

4 Experiments

Experiments evaluate the approach on Chinese-English and English-German translation, standard benchmarks, noisy inputs, and ablations. The approach improves clean-set BLEU over Transformer and remains more robust as input noise increases.

  • 4.1 Setup: Experiments use Chinese-English and English-German tasks with Transformer-based implementations and task-specific validation and test sets.Chinese-English uses LDC data and NIST tests; English-German uses WMT’14 with newstest2013 validation and newstest2014 testing.
  • 4.2 Main Results: 2.25 BLEU points average gain and up to 2.8 BLEU points on NIST03 are achieved over Trans.-Base for Chinese-English.The gains occur across validation and test sets.
  • 4.2 Main Results: 1.04 BLEU points over Trans.-Base, 1.61 over Trans.-Big, and 1.52 over RNMT+ are achieved on English-German.Comparisons use models with the same number of hidden units where applicable.
  • 4.3 Comparison to Baseline Methods: Among methods without extra corpora, the approach achieves the best result across datasets; with back-translation, it gains an additional 1-3 points over the matched baseline.All methods use the same backbone for the standard-benchmark comparison.
  • 4.4 Results on Noisy Data: On artificial noisy inputs, the approach outperforms all baselines across noise levels, with larger improvements generally appearing at higher noise fractions.Noise is created by replacing words with relevant words according to word-embedding similarity.
  • 4.4 Results on Noisy Data: Relative output BLEU drops are smaller as noise grows, and removing any training component decreases BLEU, with adversarial target input causing the largest ablation decrease of 1.87 points.The ablation compares Lclean, Lrobust, and Llm components.
  • 4.5 Ablation Studies: The best Chinese-English validation configuration uses γsrc = 0.25 and γtrg = 0.50, while larger γsrc values can damage performance.A non-zero γtrg consistently improves over γtrg = 0.

5 Related Work

Related work has addressed noisy-input robustness in NMT through black-box noise design, adversarial stability training, and white-box adversarial examples. The paper distinguishes its setting by requiring no noisy training data and targeting both clean and noisy performance.

  • Robust Neural Machine Translation: Prior NMT studies designed synthetic or natural noise using black-box methods, while adversarial stability training addressed arbitrary noise types.The cited approaches include feature-level and word-level perturbations.
  • Robust Neural Machine Translation: White-box adversarial example generation had been explored for character-level NMT, but remained comparatively limited for broader NMT settings.The paper identifies Ebrahimi et al. (2018a) as prior white-box work on character-level NMT.
  • Robust Neural Machine Translation: Unlike domain adaptation approaches that fine-tune on noisy data, this work assumes no noisy training data is available.The comparison concerns Michel and Neubig (2018).
  • Data Augmentation: The approach is also framed as data augmentation using adversarial examples, alongside prior augmentation through word dropping, replacement, and rare-word editing.These methods augment standard parallel datasets in different ways.

6 Conclusion

The paper presents doubly adversarial inputs and a white-box method for generating NMT adversarial examples. Experiments on Chinese-English and English-German tasks show improvements in translation performance and robustness.

  • 6 Conclusion: Doubly adversarial inputs are proposed to improve NMT model robustness.The paper presents this as its central approach.
  • 6 Conclusion: The paper introduces a white-box method for generating adversarial examples for NMT.The method is presented as part of the robustness approach.
  • 6 Conclusion: Experiments on Chinese-English and English-German translation tasks demonstrate improved translation performance and robustness.The conclusion reports capability on both evaluation dimensions.
  • 6 Conclusion: Future work will explore more natural adversarial examples without word replacements and more advanced defenses such as curriculum learning.These directions are explicitly identified by the paper.
Loading 1906.02443v1…