Source-linked AI summary
Data Augmentation for Low-Resource Neural Machine Translation
Marzieh Fadaee, Arianna Bisazza, Christof Monz
TL;DR
Low-resource NMT lacks sizable parallel corpora, especially diverse contexts for rare words. The paper introduces TDA, which uses monolingual language models to create paired sentences containing rare words in synthetic contexts. In simulated low-resource English–German and German–English settings, TDA improves translation quality over the baseline and back-translation.
Problem
Low-resource language pairs typically lack the numerous, diverse sentence-translation examples needed for reliable NMT parameter estimation.
Method
TDA uses language models and aligned translations to alter existing parallel-corpus sentences, generating new sentence pairs containing rare words in synthetic contexts.
Results
TDA substantially improves translation quality in simulated low-resource English–German and German–English settings and outperforms another data-augmentation technique.
Takeaways & Limitations
TDA generates more rare words during translation and is associated with higher translation quality using minor modifications to original training sentences.
Takeaways & Limitations
Augmented sentences can be grammatical without preserving the original meaning, because meaning preservation is not an objective of TDA.
Abstract
from arXiv · showhide
The quality of a Neural Machine Translation system depends substantially on the availability of sizable parallel corpora. For low-resource language pairs this is not the case, resulting in poor translation quality. Inspired by work in computer vision, we propose a novel data augmentation approach that targets low-frequency words by generating new sentence pairs containing rare words in new, synthetically created contexts. Experimental results on simulated low-resource settings show that our method improves translation quality by up to 2.9 BLEU points over the baseline and up to 3.2 BLEU over back-translation.
1 Introduction
Low-resource NMT lacks the diverse parallel examples needed for reliable parameter estimation, motivating augmentation methods. The paper proposes TDA, which alters source–target sentence pairs together to create new contexts for rare words while preserving their translation relationship.
- Computer-vision augmentation preserves labels because transformed images retain the same content, whereas modifying one side of a parallel sentence pair requires changing its translation.
- Low-resource language pairs typically lack numerous translation pairs containing words in diverse contexts, limiting NMT performance.
- TDA augments parallel-corpus training data by altering existing sentences, inspired by computer-vision data augmentation.
- TDA allows source and target sentences to change simultaneously as long as they remain translations of each other.
- The approach targets low-frequency words because rare-word parameter estimation is especially challenging in low-resource settings.
2 Translation Data Augmentation
TDA creates augmented parallel sentences by replacing common words with rare words and replacing their aligned translations, using language models to select plausible substitutions. It controls augmentation frequency and sentence-level replacement density while acknowledging that grammatical outputs need not preserve meaning.
- TDA searches for contexts where a common word can be replaced by a rare word, then substitutes its translation in the paired sentence.
- Forward and backward LSTM language models identify substitutions that produce fluent and grammatical sentences without requiring language-specific linguistic resources.
- The targeted rare-word list contains vocabulary items among the NMT system’s limited vocabulary with fewer than R corpus occurrences.
- Automatic word alignments and lexical translation probabilities select the target-language translation corresponding to each substituted source word.
- Augmentation discards substitutions without translation candidates or with language-model probability below a threshold, reducing semantically or syntactically incorrect pairs.
- Sampling limits each rare word to at most N augmentations and repeats corpus passes until no new sentences are generated.
- Grammatical augmented sentences do not necessarily preserve the original meaning, which is explicitly not an objective of the approach.
- TDAr=1 replaces one word per sentence, while TDAr≥1 permits multiple replacements at least five positions apart but carries greater noise risk.
3 Evaluation
Evaluation in a simulated low-resource setting compares TDA with baseline, back-translation, and oversampling across WMT German–English and English–German test sets. TDA performs best overall, with gains consistent across directions and substitution choices, while augmenting multiple words slightly helps despite increased noise.
- Evaluation setup: 10% of WMT15 English–German training data was sampled to simulate low-resource translation, with results reported on newstest 2014–2016.A full-data baseline was also provided for reference.
- Evaluation setup: Translation quality was measured using single-reference case-insensitive BLEU.
- Main results: TDA significantly outperforms both the low-resource baseline and back-translation across all test sets, while requiring only minor sentence modifications and no costly translation process.The comparison covers the simulated low-resource evaluation reported in Table 2.
- Main results: Improvements are consistent across German–English and English–German, regardless of whether rare-word substitutions begin on the source or target side.
- Ablations and analysis: Altering multiple words per sentence performs slightly better than altering one, although the resulting augmented sentences are likely noisier.
- Ablations and analysis: Sampled data replication outperforms the baseline and back-translation, but TDA remains the best-performing system overall.The effect is not reproduced by simply continuing baseline training for up to 50 epochs.
- Output analysis: Augmented-data systems generate translations closer to reference length: baseline 0.88, TDA variants 0.95 and 0.94.The authors attribute this to generating rare-word translations that the baseline leaves untranslated.
4 Analysis of the Results
TDA improves rare-word handling by creating new contexts, increasing rare words generated during translation and raising attention to rare source words, while introducing some erroneous augmented data.
- Rare-word generation: TDA enables the model to generate the rare translation “centimetres,” which the baseline omits, producing a more fluent sentence.Augmented training examples provide new contexts for the rare target word, including some nonsensical sentences.
- Rare-word generation: Three times more rare words in the reference vocabulary appear in TDA outputs than in baseline outputs.Most targeted words were no longer rare after sufficient augmentation.
- Source-side attention: 8.8% higher average attention scores for rare source words are observed with TDA than with the baseline in English–German translation.The authors suggest this reflects more accurate rare-word representations and greater confidence attending to them.
- Error analysis: Incorrect augmented sentences can result from low-probability substitutions suggested by the English language model.One documented error is an ungrammatical sentence after substituting “of” with “yearly.”
- Error analysis: Translation-selection errors can arise from the German language model or from the lack of a suitable bitext-derived translation, though the noise has limited effect on NMT.The example uses “betraut,” an acceptable translation that would require rephrasing the German sentence.
5 Conclusion
The paper proposes TDA for low-resource NMT, using monolingual language models to create sentence pairs with rare words in synthetic contexts. The approach increases rare-word generation and improves translation quality in simulated low-resource settings.
- Contribution: TDA augments low-resource NMT training data with rare words in new, synthetically created contexts.The method leverages language models trained on large amounts of monolingual data.
- Findings: TDA generates more rare words during translation and leads to higher translation quality.The conclusion reports substantial improvements in simulated English–German and German–English settings.
- Comparison: TDA outperforms another recently proposed data augmentation technique in simulated low-resource settings.