Source-linked AI summary
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
Xinyi Wang, Hieu Pham, Zihang Dai, Graham Neubig
TL;DR
NMT data augmentation must expand limited empirical training data without introducing semantic inconsistencies or brittle noise. The paper formulates augmentation as an optimization problem, derives a generic analytic solution, and proposes SwitchOut; across translation tasks at different scales, SwitchOut improves performance and is efficient to implement.
Problem
NMT augmentation is difficult because sentence modifications can change semantics or require corresponding translation changes, while empirical training data may insufficiently cover the valid data space.
Method
The paper formulates augmentation-policy design as an optimization problem balancing smoothness and diversity, then derives an analytic framework that motivates SwitchOut.
Results
SwitchOut delivers significant improvements over the best non-SwitchOut baseline in 4 of 6 settings, with all WMT 15 en-de gains significant at p < 0.0002.
Takeaways & Limitations
SwitchOut is efficient and easy to implement, with improvements reported across translation tasks at different scales.
Takeaways & Limitations
The efficient sampling procedure approximates the true distribution by altering relative probabilities across different Hamming distances.
Abstract
from arXiv · showhide
In this work, we examine methods for data augmentation for text-based tasks such as neural machine translation (NMT). We formulate the design of a data augmentation policy with desirable properties as an optimization problem, and derive a generic analytic solution. This solution not only subsumes some existing augmentation schemes, but also leads to an extremely simple data augmentation strategy for NMT: randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies. We name this method SwitchOut. Experiments on three translation datasets of different scales show that SwitchOut yields consistent improvements of about 0.5 BLEU, achieving better or comparable performances to strong alternatives such as word dropout (Sennrich et al., 2016a). Code to implement this method is included in the appendix.
1 Introduction and Related Work
NMT data augmentation is difficult because sentence changes can alter meaning or require coordinated translation changes, while indiscriminate modifications can introduce harmful noise. The paper addresses this gap by deriving a simple optimization-based policy and proposing SwitchOut, which replaces source- and target-side words independently.
- Data augmentation benefits learning algorithms through increased training-set quantity and diversity, although generated examples may be lower quality.
- Small sentence modifications can change semantics or require corresponding translation changes, making simple NLP augmentation equivalences difficult.
- Indiscriminate NMT data modifications can introduce noise that makes translation systems brittle.
- Existing NMT augmentation methods primarily use back-translation or word replacement, with back-translation vulnerable to errors in initial models.
- SwitchOut independently replaces source and target words with uniformly sampled words from their respective vocabularies.
2 Method
The method formulates data augmentation as maximizing smoothness and diversity, yielding a generic analytic framework and the SwitchOut algorithm for independently replacing source- and target-side words.
- Augmentation framework: The framework models augmentation as a distribution q over expanded training pairs to address the mismatch between empirical and true data supports.Augmented training supplies additional pairs beyond the observed bootstrap distribution.
- Augmentation framework: The augmentation objective combines similarity to observed pairs with entropy, balancing smoothness against diversity through τ.The similarity term favors cases resembling observed pairs, while entropy broadens the augmented support.
- Generic solution: The resulting maximum-entropy problem has an analytic solution that is agnostic to the similarity function and encompasses word dropout and RAML.The framework therefore provides a common interpretation of these existing augmentation schemes.
- SwitchOut: SwitchOut augments both source and target sentences, factoring their similarity measures so the two augmented sentences can be sampled independently.The method uses separate domain-specific similarity functions and temperature hyper-parameters.
- SwitchOut: SwitchOut samples a replacement count and independently replaces each sentence word with probability b_n/|s| by a uniformly chosen different vocabulary word.The sampling procedure is designed for parallelized implementation across sentence positions and batches.
- SwitchOut: The practical sampling procedure preserves equal probabilities for sentences at the same Hamming distance but approximates the intended distribution across different distances.Despite this approximation, the procedure is easier to implement while achieving good performance.
3 Experiments
Experiments evaluate SwitchOut across three translation tasks and compare it with Transformer, word dropout, RAML, and back translation. Results indicate significant gains in many settings, complementary benefits with RAML and back translation, and stronger improvements for test sentences farther from training data.
- Datasets and procedures: SwitchOut is benchmarked on IWSLT 2015 en-vi, IWSLT 2016 de-en, and WMT 2015 en-de using word-based translations.The translation model is a Transformer, with hyperparameters tuned separately for each dataset.
- Baselines: SwitchOut is compared against Transformer, source-side word dropout, target-side RAML, and, on en-de, back translation.The word-dropout baseline uses λword = 0.1.
- SwitchOut comparisons: 4 of 6 settings show significant improvements over the best baseline without SwitchOut, while the remaining two differences are not statistically significant.On WMT 2015 en-de, all gains over the best baseline are significant with p < 0.0002.
- SwitchOut comparisons: SwitchOut on the source achieves gains as large as target-side RAML, and combining SwitchOut with RAML yields further improvements.Table 1 reports median BLEU scores across multiple runs, with significance markers for comparisons against the best result without SwitchOut.
- SwitchOut vs. back translation: Back translation provides a less significant gain than SwitchOut on en-de, while applying SwitchOut to back-translated data further improves BLEU.Table 2 compares back translation with SwitchOut and their combination using median scores from four runs.
- Effects of τx and τy: When τy is fixed, the best dev-set performance is always achieved with a non-zero τx.Figure 1 reports dev BLEU scores across τx and τy on all three tasks.
- Where SwitchOut helps most: SwitchOut improves increasingly more as nearest-neighbor WER increases, indicating larger gains for test sentences farther from training sentences.Figure 2 orders test sentences by WER to their nearest training neighbor and plots the BLEU gain of RAML+SwitchOut over RAML.
4 Conclusion
The paper derives an optimization-based framework for designing data augmentation algorithms and uses it to inspire SwitchOut. SwitchOut improves translation performance across tasks of different scales while remaining efficient and easy to implement.
- Conclusion: The proposed optimization framework subsumes several existing augmentation schemes and inspires the novel SwitchOut method.The framework is formulated by optimizing a data augmentation policy.
- Conclusion: SwitchOut delivers improvements across translation tasks at different scales.The conclusion also characterizes SwitchOut as efficient and easy to implement.
- Conclusion: SwitchOut’s efficiency and ease of implementation support its potential for wide application.
A.1 Word Dropout as a Special Case
The framework recovers word dropout by introducing an untrained null token and defining neighborhoods of sentences formed by replacing words with that token. Compared with SwitchOut, word dropout samples from a smaller support, concentrating probability mass while preserving smoothness.
- For a sequence x, the neighborhood N(x) contains x and sentences formed by replacing a few words with ⟨null⟩.
- The augmentation policy decomposes into independent sampling of augmented source and target sentences, and word dropout instantiates both terms symmetrically.
- Word dropout corresponds to independently dropping words with a Bernoulli distribution parameterized by λ_word.
- SwitchOut samples from a support of V^|x|, whereas word dropout concentrates all probability mass in the much smaller N(x), yielding lower entropy.
- Both methods are exponentially less likely to diverge from x, satisfying the framework’s smoothness desideratum.
A.2 RAML as a Special Case
RAML can be represented within the proposed augmentation framework by treating its reward-aware target distribution as an augmentation distribution. This representation restricts augmentation to pairs whose source sentence remains in the empirical set.
- RAML trains the model distribution p_θ(Y | x) to match a reward-aware target distribution p_RAML(Y | x, y).
- The RAML objective minimizes the expected KL divergence between p_RAML and p_θ.
- The marginalized target distribution p_RAML(Y) belongs to the augmentation-distribution family considered by the framework.
- RAML is a special case in which the similarity function assigns reward-based scores when bx = x and excludes other source sentences.
- Practically, RAML considers only pairs with source sentences drawn from the empirical set for data augmentation.
A.3 Datasets Descriptions
The experiments benchmark SwitchOut on three standard, word-based translation tasks spanning different dataset scales. The datasets include IWSLT English–Vietnamese, IWSLT German–English, and WMT English–German.
- Table 3 summarizes the statistics of the experimental datasets.
- The WMT 15 en-de dataset is one order of magnitude larger than the IWSLT 16 de-en and IWSLT 15 en-vi datasets.
- For en-vi, the study uses data pre-processed by Luong and Manning (2015).
- For en-de, the study uses Luong et al. (2015) preprocessing, with newstest2014 for validation and newstest2015 for testing.
A.4 Hyper-parameters
The experiments use Adam optimization, gradient clipping, task-specific learning-rate schedules, and 100,000 training steps. Multiple GPUs are used, with update counts adjusted for the de-en and en-vi experiments.
- Table 4 specifies the hyper-parameters used in the experiments.
- All models are initialized uniformly at random and trained with Adam, with gradients clipped at the Table 4 threshold.
- For WMT en-de, the experiments use the legacy learning-rate schedule specified by Vaswani et al. (2017).
- For de-en and en-vi, the learning rate starts at 0.001 and decreases by 0.97 every 1000 steps from step 8000.
- All models are trained for 100,000 steps, with one checkpoint selected during training.
- For de-en and en-vi, using n GPUs with n ∈ {1, 2, 4} reduces parameter updates to 10^5/n, which the authors found sufficient for convergence.
A.5 Source Code for Sampling in TensorFlow
The TensorFlow implementation samples corrupted sentence batches by choosing corruption counts, positions, and replacement values while preserving special tokens. It accepts padded sentence tensors and vocabulary metadata, returning corrupted sentences.
- A.5 Source Code for Sampling in TensorFlow: The hamming_distance_sample function samples a batch of corrupted examples from input sentence tensors.The inputs are shaped [batch_size, n_steps], and the output is a batch of corrupted sentences.
- A.5 Source Code for Sampling in TensorFlow: The sampler uses tau to compute logits over the number of time steps before sampling corruption counts.The implementation forms logits from -tf.range(tf.to_float(n_steps)) * tau.
- A.5 Source Code for Sampling in TensorFlow: The implementation samples corrupted positions after determining how many words to corrupt in each sentence.The code separately marks the stages for sampling sentence-level corruption counts and corrupted positions.
- A.5 Source Code for Sampling in TensorFlow: Beginning-of-sentence and end-of-sentence tokens are excluded from replacement, and padding is identified during masking.The implementation compares sentence entries with bos_id, eos_id, and pad_id while explicitly protecting <bos> and <eos>.
- A.5 Source Code for Sampling in TensorFlow: Replacement values are sampled as integer vocabulary indices for the sentence tensor dimensions.The sampling range uses shape [batch_size, n_steps], with values from 1 through vocab_size.
A.6 Source Code for Sampling in PyTorch
The PyTorch implementation samples corrupted sentence batches by choosing corruption counts, positions, and replacement values. It masks beginning, ending, and padding tokens during the sampling process.
- A.6 Source Code for Sampling in PyTorch: The PyTorch routine samples a batch of corrupted examples from input sentence tensors.The inputs and corrupted outputs are represented as tensors with shape [batch_size, n_steps].
- A.6 Source Code for Sampling in PyTorch: The sampler masks beginning-of-sentence, end-of-sentence, and padding positions before selecting corruption locations.The mask combines equality checks against bos_id, eos_id, and pad_id.
- A.6 Source Code for Sampling in PyTorch: The implementation first samples the number of words to corrupt for each sentence, then samples corrupted positions.The code marks these as successive sampling stages.
- A.6 Source Code for Sampling in PyTorch: Masked positions receive -float("inf") before position sampling, preventing them from being selected.The masking operation is applied to the sentence tensor before sampling corrupted positions.
- A.6 Source Code for Sampling in PyTorch: Sampled corrupted values are added to the input sentences to produce the corrupted outputs.The implementation comments identify the replacement values as values added to sents.