Source-linked AI summary
AdvAug: Robust Adversarial Augmentation for Neural Machine Translation
Yong Cheng, Lu Jiang, Wolfgang Macherey, Jacob Eisenstein
TL;DR
NMT systems can lose performance under input perturbations, while augmentation faces the challenge of keeping noisy translation pairs semantically valid. AdvAug trains on embeddings of virtual sentences sampled from adversarial and authentic vicinity distributions. Across three translation benchmarks, it improves performance, with gains of up to 4.9 BLEU points over the Transformer.
Problem
NMT performance can drop sharply under meaning-preserving input perturbations, and augmentation methods face the challenge of maintaining semantic validity while addressing insufficient training diversity and overfitting.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Compared with baseline methods without extra corpora, AdvAug improves over state-of-the-art models on standard datasets.
Source: §4.2 · p. 7The approach assumes that all adversarial examples originating from a training pair are translated into the same target sentence.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
In this paper, we propose a new adversarial augmentation method for Neural Machine Translation (NMT). The main idea is to minimize the vicinal risk over virtual sentences sampled from two vicinity distributions, of which the crucial one is a novel vicinity distribution for adversarial sentences that describes a smooth interpolated embedding space centered around observed training sentence pairs. We then discuss our approach, AdvAug, to train NMT models using the embeddings of virtual sentences in sequence-to-sequence learning. Experiments on Chinese-English, English-French, and English-German translation benchmarks show that AdvAug achieves significant improvements over the Transformer (up to 4.9 BLEU points), and substantially outperforms other data augmentation techniques (e.g. back-translation) without using extra corpora.
1 Introduction
NMT systems can be vulnerable to meaning-preserving input perturbations, while training data diversity and semantic validity constrain existing augmentation approaches. AdvAug trains on embeddings of virtual sentences sampled from adversarial and authentic vicinity distributions, and reports improvements across three translation benchmarks.
1 Introduction
- Meaning-preserving input perturbations can sharply reduce NMT performance and translation quality, limiting use in more sensitive scenarios.The introduction illustrates this with a synonym substitution that changes the Transformer’s translation to a contradictory result.Source: §1 · p. 1
- Insufficient training-set diversity and model overfitting contribute to poor generalization, while noisy augmentation must preserve the semantics of translation pairs.Continuous noise constrained by an L2-norm ball does not guarantee semantic preservation; Cheng et al. (2019) generated adversarial sentences that the authors describe as unnatural and suboptimal.Source: §1 · p. 1
- AdvAug samples virtual adversarial sentences from a new vicinity distribution and trains on their embeddings, alongside embeddings from an authentic-data vicinity distribution.The approach also uses a mini-batch augmentation algorithm for sequence-to-sequence NMT training.Source: §1 · p. 2
- Up to 4.9 BLEU points over the Transformer and up to 3.3 BLEU points over Cheng et al. (2019) are reported across three translation benchmarks.AdvAug also yields better performance than widely used data augmentation methods without extra corpora.Source: §1 · p. 2
2 Background
The background introduces the encoder-decoder formulation and empirical-risk objective for NMT, then reviews adversarial examples and mixup as augmentation approaches. It frames adversarial examples as perturbations constrained by semantic similarity and mixup as interpolation between sampled examples.
Neural Machine Translation
- NMT models estimate translation probability P(y|x; θ) with an encoder-decoder, where the decoder conditions on prior target tokens and the encoded source.The paper denotes sentence feature vectors or word embeddings by e(x).Source: §2 · p. 2
- Standard NMT training minimizes empirical risk over the parallel corpus using cross-entropy on position-wise predictions and one-hot target labels.The training distribution is described as the empirical distribution of the corpus, with δ denoting the Dirac delta function.Source: §2 · p. 2
Generating Adversarial Examples for NMT
- Cheng et al. (2019) generate adversarial examples by learning perturbations jointly with the NMT model, with semantic similarity constrained by an upper bound.Their examples are used to attack and defend the model, and the robustness loss is combined with clean loss for training.Source: §2 · pp. 2–3
Data Mixup
- Mixup trains on linear interpolations of randomly sampled image pairs and labels, with the mixing weight drawn from Beta(α, α).As α approaches zero the mixture stays near one sampled pair; as α grows it approaches the interpolation midpoint.Source: §2 · p. 3
3 Approach: AdvAug
AdvAug trains NMT over virtual examples drawn from adversarial and authentic vicinity distributions around training examples. Its central component interpolates embeddings of same-origin adversarial examples, while a complementary distribution mixes authentic examples from different origins.
3 Approach: AdvAug
- Same-origin adversarial examples are interpolated in embedding space to create virtual examples that share a translation target; this distribution is identified as the most important factor for translation quality.The authentic-data distribution mixes embeddings of observed examples from different origins and is complementary, though it yields only marginal improvements alone.Source: §3 · pp. 3–4
3.1 Padv for Adversarial Data
- AdvAug constructs virtual adversarial pairs by interpolating aligned embeddings from two adversarial examples sampled from the same training pair's adversarial set.The interpolation coefficient is sampled from a Beta distribution; all adversarial examples from one origin are supposed to share a target, allowing virtual pairs to use the same soft target.Source: §3.1 · p. 4
- The adversarial loss matches word-level model predictions on virtual examples to a fixed copy of the model's predictions on observed examples.The soft target lets virtual adversarial examples learn from output distributions over large vocabularies; training samples one adversarial pair per example per mini-batch and uses curriculum-based importance sampling.Source: §3.1 · pp. 4–5
3.2 Paut for Authentic Data
- Paut defines authentic-data vicinal loss over observed examples and their embedding interpolations, with λ set either to 1.0 or sampled from a Beta distribution.The λ = 1.0 case recovers clean-data loss, while Beta-sampled λ yields mixup loss with interpolated model-output targets.Source: §3.2 · p. 5
3.3 Training
- The training objective minimizes the sum of the authentic-data and adversarial-data losses, computed from mini-batches of interpolated virtual examples.For Padv, a pair is sampled per training example and the distribution is covered over multiple epochs; Paut uses within-batch pair sampling.Source: §3.3 · p. 5
4 Experiments
Experiments evaluate AdvAug on Chinese-English, English-French, and English-German translation tasks. The authors report gains over Transformer and other baselines, with improvements in noisy-input robustness and resistance to overfitting.
4.1 Setup
- The evaluation uses BLEU on Chinese-English, English-French, and English-German tasks, with training sets of 1.2M, 0.23M, and 4.5M sentence pairs, respectively.The reported tasks use the LDC, IWSLT 2016, and WMT14 corpora.Source: §4.1 · p. 6
4.2 Main Results
- AdvAug outperforms Transformer by 3.34 BLEU points on English-French and 2.27 BLEU points on English-German.The authors also report that combining Ladv and Laut performs best on both tasks.Source: Table 2 · p. 7, §4.2 · p. 7
- Combining Laut and Ladv yields the best Chinese-English performance, with gains of up to 4.92 BLEU points on MT05; Ladv is more important.Both losses improve the Transformer, while Lmixup is only slightly better than the standard empirical loss.Source: §4.2 · p. 7
- AdvAug significantly improves over state-of-the-art baselines without extra corpora, and adding back-translated data yields further gains.The authors report no further gain from directly incorporating adversarial examples without embedding interpolation over Cheng et al. (2019).Source: §4.2 · p. 7
4.4 Robustness to Noisy Inputs and Overfitting
- For an adversarial input, AdvAug uses virtual-sentence embeddings containing “danshi” or its synonym “dan,” which the authors say encourages their embeddings to move closer.Source: Table 3 · p. 8, §4.4 · p. 9
- AdvAug shows higher robustness than all baseline methods across the tested noise levels.The noisy validation sets were created by replacing a word in each sentence with a relevant alternative.Source: Table 5 · p. 8, §4.4 · p. 8
4.3 Effect of α
- Laut and Ladv perform better at α=8, while Lmixup performs slightly better at α=4 but takes 90K iterations to converge versus 20K at α=0.2.At α=32, Lmixup fails model training.Source: §4.3 · p. 8, Table 4 · p. 8
4.4 Robustness to Noisy Inputs and Overfitting
- On the Chinese-English validation set, Lclean’s BLEU score peaks at about 20K iterations and then the model starts overfitting; proposed training losses show no significant regression even after 100K iterations.Source: §4.4 · p. 8, Figure 2 · p. 8
5 Related Work
Prior NMT augmentation approaches use extra corpora or original parallel data, while robustness research has also explored adversarial examples and improved representations. AdvAug differs by augmenting adversarial-example representations through interpolation and reports gains over back-translation without extra monolingual corpora.
Data Augmentation
- Prior NMT data augmentation draws either on extra corpora or on original parallel corpora, while mixup has also been used for semi-supervised learning and noisy data.Source: §5 · p. 9
- AdvAug augments adversarial-example representations in sequence-to-sequence training and outperforms back-translation without extra monolingual corpora.The authors also report better performance when additional monolingual corpora are included.Source: §5 · p. 9
Robust Neural Machine Translation
- Robustness studies have augmented training data with adversarial examples and also explored better input representations and adversarial regularization.Source: §5 · p. 9
6 Conclusion
The conclusion presents AdvAug as an NMT training method that samples virtual sentences from vicinity distributions over interpolated embeddings. Experiments on three language-pair tasks demonstrate improved translation performance and robustness.
6 Conclusion
- AdvAug augments NMT training with a vicinity distribution over interpolated embeddings of adversarial examples.Source: §6 · p. 9
- The training algorithm samples virtual sentences from both the adversarial-example vicinity distribution and a mixup-like distribution over observed examples.Source: §6 · p. 9
- Experiments on Chinese-English, English-French, and English-German demonstrate improved translation performance and robustness.Source: §6 · p. 9