Source-linked AI summary
Meta-Learning for Low-Resource Neural Machine Translation
Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, Victor O. K. Li
TL;DR
Low-resource NMT suffers when parallel data is limited, and existing multilingual approaches do not always provide a shared space across languages. The paper extends MAML by learning an initialization from high-resource language tasks and using universal lexical representations to address vocabulary mismatch. MetaNMT significantly outperforms multilingual transfer learning across the evaluated language pairs, while its performance depends on validation-task choice and source-task selection.
Problem
Low-resource NMT can overfit with limited data, while multilingual methods face mismatched input and output spaces across languages.
Method
The paper extends MAML by treating language pairs as tasks, learning an initialization from high-resource languages, and incorporating universal lexical representation.
Results
MetaNMT significantly outperforms multilingual transfer learning across all considered language pairs.
Takeaways & Limitations
The approach supports competitive low-resource NMT through fast adaptation from high-resource language-pair tasks.
Takeaways & Limitations
Validation-task choice has a nonnegligible impact on final performance, and its relationship with task similarity requires further investigation.
Abstract
from arXiv · showhide
In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML) for low-resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation~\citep{gu2018universal} to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach~\citep{zoph2016transfer} and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT'16 by seeing only 16,000 translated words (~600 parallel sentences).
1 Introduction
Low-resource NMT can lag behind conventional systems, motivating methods that transfer knowledge from high-resource languages. This paper frames translation as meta-learning and reports stronger performance across target language pairs, especially with fewer examples.
- Low-resource vanilla NMT often lags behind conventional phrase-based translation systems.
- Multilingual NMT and transfer learning exploit knowledge from high-resource language pairs for low-resource translation.Transfer learning pretrains on a high-resource pair before fine-tuning on the target pair.
- Meta-learning treats language pairs as separate tasks and learns parameters that facilitate fast adaptation to a new language pair.The approach extends MAML and uses universal lexical representation to handle mismatched input and output spaces.
- The evaluation uses 17 Europarl languages and Russian as source tasks, with five target languages translated into English.The target tasks are Romanian, Latvian, Finnish, Turkish, and Korean.
- The meta-learning approach outperforms multilingual translation across all target language pairs, with the performance gap growing as training examples decrease.Experiments use up to 160k target-task tokens.
2 Background
Low-resource NMT is vulnerable to overfitting and limited data, while multilingual transfer methods seek to share information across language pairs. The paper introduces MAML-based learning of parameter initializations for fast adaptation in this setting.
- NMT can overfit easily and perform poorly when training data is limited.
- Low-resource translation approaches use unlabeled monolingual data or share knowledge between low- and high-resource language pairs.
- Multilingual NMT can improve low-resource translation by sharing embedding information through a universal lexical representation.
- Previous multilingual NMT methods do not necessarily produce a universal input and model space, especially in very low-resource settings.
- Meta-learning learns to adapt rapidly to new training data, including by learning parameter initializations for fast adaptation.This paper extends MAML in a multilingual scenario for low-resource NMT.
3 Meta Learning for Low-Resource Neural Machine Translation
MetaNMT learns parameter initializations that enable rapid adaptation to low-resource language pairs. It simulates language-specific learning on high-resource source tasks, evaluates the adapted parameters, and addresses cross-language vocabulary mismatch with universal lexical representations.
- Meta-learning formulation: MAML treats language pairs as separate tasks and learns an initialization from source tasks for fast adaptation to a target language pair.The target model is trained from the learned initialization using only a small number of examples.
- Language-specific learning: Language-specific learning maximizes the data log-posterior while penalizing deviation from the initial parameters, helping reduce overfitting with scarce training data.In practice, optimization uses gradient-based updates and early stopping after only a few steps.
- Meta-learning procedure: Each meta-learning episode samples one source task, uses one data subset for simulated learning, and evaluates the adapted parameters on an independently sampled subset.The evaluation gradient updates the meta-model, and multiple episodes may be aggregated before updating it.
- Comparison with prior approaches: Unlike multilingual and transfer learning, MetaNMT explicitly simulates learning on low-resource target tasks rather than only optimizing a shared or pretrained model.The proposed framework incorporates the target-task learning process through repeated simulation during meta-training.
- Unified lexical representation: Universal lexical representation dynamically builds language-specific vocabularies from shared parameters, allowing MetaNMT to handle differing input and output vocabularies.The representation uses universal and key embedding matrices with a language-specific convex combination, while shared matrices remain fixed during meta-learning.
- Unified lexical representation: The universal embedding matrix is not updated during small-corpus fine-tuning; a separate parameter estimates language-specific embedding changes.This avoids changing embeddings for tokens that are absent from the limited target corpus.
4 Experimental Settings
The experiments evaluate low-resource translation across five diverse target languages using multilingual source tasks, strict validation, and controlled training settings. They vary source-task sets, target-data sizes, model components fine-tuned, and multilingual lexical preprocessing.
- Target Tasks: Five target languages—Romanian, Latvian, Finnish, Turkish, and Korean—are evaluated using official train, development, and test splits.Romanian uses WMT’16, Latvian, Finnish, and Turkish use WMT’17, and Korean uses the Korean Parallel Dataset.
- Source Tasks: Eighteen source languages from Europarl and WMT are combined in varying task sets to study their effect on adaptation.The source languages comprise 17 Europarl languages plus Russian from WMT.
- Validation: Meta-learning uses either Romanian-English or Latvian-English as validation while testing generalization on the remaining target tasks.This evaluates the strict setting where target tasks are unknown during both training and model selection.
- Preprocessing and Model: The Transformer uses universal lexical representations, with 40,000 BPE subwords per language and multilingual word vectors aligned using MUSE.The universal embedding matrix uses multilingual vectors for the 20,000 most frequent English words.
- Experimental Protocol: Target tasks contain 4k, 16k, 40k, or 160k English tokens, with five random samples per experiment and average scores plus standard deviations reported.Fine-tuning uses validation-based early stopping before test-set evaluation; 16k tokens are the default setting.
- Fine-tuning Strategies: Fine-tuning compares updating all Transformer modules, updating embedding plus encoder while freezing the decoder, and updating only the embedding.The three strategies are called all, emb+enc, and emb.
5 Results
Experiments show that MetaNMT consistently outperforms multilingual transfer learning across low-resource target languages, with advantages increasing as training data decreases. Additional analyses examine source-task selection, training stability, and fine-tuning effects.
- Multilingual Transfer Learning: MetaNMT significantly outperforms multilingual transfer learning across Ro-En, Lv-En, Fi-En and Tr-En target tasks.The comparison uses 16k English tokens per target task and remains consistent regardless of the validation task used for early stopping.
- Statistical Machine Translation: 4.79(±0.234) BLEU is achieved by phrase-based MT on Ro-En with 16,000 target tokens, exceeding standard NMT's 0 BLEU but remaining below multi-NMT and MetaNMT.
- Training Set Size: The gap between MetaNMT and transfer learning grows as the target training set shrinks, indicating greater robustness in extremely low-resource settings.
- Impact of Source Tasks: Up to 2× improvement occurs when increasing source tasks from one to 18, although the benefit and preferred source languages vary by target language.Ro-En benefits more from {Es, Fr, It, Pt} than from {De, Ru}, while the other target tasks show the opposite pattern.
- Training Curves: Multilingual transfer learning saturates and degrades from source-task overfitting, whereas MetaNMT continues improving without degradation.The meta-objective favors parameters suitable for fine-tuning on target tasks rather than solely solving source tasks.
- Sample Translations: After about 600 sentence pairs, the meta-learned model rapidly learns Turkish-to-English token reordering and produces better translations.Without Tr-En examples, the model produces word-by-word translations; similar behavior appears for Ko-En.
6 Conclusion
The paper introduces MetaNMT, which combines multilingual high-resource language tasks, MAML-based meta-learning, and universal lexical representations for low-resource translation. Across five target languages, it outperforms multilingual transfer learning and offers a framework for incorporating additional data and translation systems.
- 6 Conclusion: MetaNMT extends MAML to multilingual low-resource NMT and uses universal lexical representation for languages without a shared vocabulary.
- 6 Conclusion: Across 18 high-resource source tasks and five low-resource target tasks, MetaNMT significantly outperforms multilingual transfer learning across all considered language pairs.
- 6 Conclusion: The framework can incorporate source- and target-side monolingual corpora and accommodate existing and future NMT systems.