Source-linked AI summary
Transfer Learning for Low-Resource Neural Machine Translation
Barret Zoph, Deniz Yuret, Jonathan May, Kevin Knight
TL;DR
Low-resource NMT struggles because neural methods require abundant parallel data and learn poorly from scarce examples. The paper transfers parameters from a high-resource parent model to initialize and constrain a low-resource child model, improving NMT and SBMT rescoring results. Across four language pairs, transfer improves baseline NMT by 5.6 BLEU on average, while the final system reaches competitive SBMT performance and improves SBMT rescoring.
Problem
NMT is effective with large datasets but performs poorly for low-resource languages where parallel data is scarce.
Method
The method trains a high-resource parent NMT model, transfers its parameters to a low-resource child model, and constrains training by fixing selected parameters.
Results
Transfer improves baseline NMT models by an average of 5.6 BLEU across four low-resource language pairs, and rescoring improves SBMT by 1.1–1.6 BLEU points.
Takeaways & Limitations
Transfer NMT approaches strong SBMT performance, beats it on one language pair, and consistently improves SBMT output when used for rescoring.
Takeaways & Limitations
The optimal parameter-freezing configuration is likely language- and corpus-dependent.
Abstract
from arXiv · showhide
The encoder-decoder framework for neural machine translation (NMT) has been shown effective in large data scenarios, but is much less effective for low-resource languages. We present a transfer learning method that significantly improves Bleu scores across a range of low-resource languages. Our key idea is to first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. Using our transfer learning method we improve baseline NMT models by an average of 5.6 Bleu on four low-resource language pairs. Ensembling and unknown word replacement add another 2 Bleu which brings the NMT performance on low-resource machine translation close to a strong syntax based machine translation (SBMT) system, exceeding its performance on one language pair. Additionally, using the transfer learning model for re-scoring, we can improve the SBMT system by an average of 1.3 Bleu, improving the state-of-the-art on low-resource machine translation.
1 Introduction
NMT performs well with abundant data but learns poorly from scarce parallel text, making vanilla NMT weak for low-resource languages. The paper introduces transfer learning to improve these systems and compares them with strong SBMT baselines.
- NMT is data-hungry and learns poorly from low-count events, limiting its suitability for low-resource languages.
- SBMT strongly outperforms attention-based NMT on four low-resource language pairs.
- The method trains a high-resource parent model, then uses its parameters to initialize and constrain a low-resource child model.
- Transfer learning improves baseline NMT models by an average of 5.6 BLEU across four low-resource language pairs.
- NMT can improve SBMT output when used as a rescoring model.
2 NMT Background
The NMT system uses an encoder-decoder architecture in which recurrent networks represent the source sentence and generate the target sentence.
- The encoder converts a source sentence into a dense, fixed-length vector.
- The decoder converts the encoded vector into a target sentence.
- The paper uses a two-layer encoder-decoder with LSTM units, trained by maximum likelihood with back-propagation.
3 Transfer Learning
Transfer learning initializes a low-resource NMT model from a high-resource parent model rather than random weights, then preserves useful shared parameters during child training. The approach is motivated by transferable representations and produces large BLEU gains across low-resource languages.
- Transfer learning uses knowledge from a learned task to improve performance on a related task, typically reducing required training data.
- The method trains a bilingual parent model on abundant data and initializes a child model trained on scarce bilingual data.
- The parent model supplies a strong prior, while parameters likely to transfer are fixed during child-model training.
- Transfer models produce large BLEU increases and beat a strong SBMT baseline on one of four language pairs.
4 Experiments
Experiments evaluate transfer learning for Hausa, Turkish, Uzbek, and Urdu into English against NMT and SBMT baselines. Transfer substantially improves standalone NMT, while ensembling, unknown word replacement, and rescoring provide additional gains.
- Experimental Setup: The low-resource experiments use French as the parent source language, four child source languages, and English as the target.The parent French-English model is trained on substantially more English-token data than the child language pairs.
- Transfer Results: 5.6 average BLEU improvement over baseline NMT is obtained with transfer learning across four low-resource language pairs.The transfer condition is compared with baseline NMT without transfer.
- Transfer Results: 7.5 average BLEU improvement is achieved after ensembling 8 models and applying unknown word replacement.These enhancements further improve the transfer-learning BLEU scores.
- Transfer Results: Transfer-learning NMT reaches competitive scores and beats the SBMT system for one of the four language pairs.The experiments use a strong string-to-tree statistical machine translation baseline.
- Re-scoring Results: Transfer NMT rescoring of SBMT n-best lists yields 1.1–1.6 BLEU points above the strong SBMT system.The n-best lists contain 1000 candidates, and transfer models outperform rescoring with a neural language model or non-transfer NMT.
5 Analysis
The analysis examines how parent-language choice, parameter freezing, initialization, and parent-model objectives affect transfer-learning performance. Results show that closer parent-child language matches help, selective freezing is optimal, transfer improves development generalization, and bilingual parent training matters.
- 5.1 Different Parent Languages: French transfer raises Spanish-English BLEU from 16.4 to 31.0, compared with 29.8 using German transfer.French is reported as a better parent for Spanish, possibly because of greater language similarity.
- 5.2 Effects of having Similar Parent Language: A closely related synthetic parent-child pair yields a 6.7 BLEU improvement, versus 4.3 BLEU with an unrelated French-parent and Uzbek-child pair.The synthetic French’ child preserves French distributional and relational properties while changing vocabulary spellings.
- 5.3 Ablation Analysis: The optimal French-English to Uzbek-English setting trains every child component except the input and output target embeddings.The authors caution that the best freezing configuration may depend on the language and corpus; Turkish benefits slightly from additionally freezing target attention parameters.
- 5.4 Learning Curve: Transfer and non-transfer models reach similar training perplexities, but the transfer model achieves much better development perplexity.The authors interpret this as transfer guiding optimization toward a solution that generalizes better despite similar training-set performance.
- 5.5 Dictionary-Based Initialization: Dictionary-based Uzbek-to-French embedding assignment reaches 25 development perplexity at epoch 2 rather than epoch 4, although final perplexities are similar.The model can untangle the initial random Uzbek/French word-type mapping without dictionary assistance.
- 5.6 Different Parent Models: English-only parent models do not match the transfer behavior of bilingual training, indicating that gains use translation parameters learned from large bilingual text.The comparison includes English-English copying, EngPerm-English unscrambling, and an English language-model parent.
6 Conclusion
Transfer learning substantially improves NMT for low-resource languages and brings it close to a strong SBMT system, exceeding SBMT on Hausa-English. Transfer NMT also consistently improves SBMT when used for rescoring.
- 6 Conclusion: Transfer learning improves low-resource NMT by a large margin and brings performance close to a strong SBMT system, exceeding it on Hausa-English.The conclusion reports this outcome across the low-resource experiments.
- 6 Conclusion: Transfer NMT consistently and significantly improves state-of-the-art SBMT systems when used for rescoring.The authors suggest that more similar parent languages could further improve results if suitable data are available.