Source-linked AI summary
Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation
Toan Q. Nguyen, David Chiang
TL;DR
NMT performs poorly on very low-resource language pairs, and conventional transfer does not exploit source-vocabulary overlap between related languages. The paper combines transfer learning with BPE-based vocabulary sharing, obtaining larger gains over BPE, including up to 4.3 BLEU. Further investigation is needed to determine whether the approach works beyond the studied language types.
Problem
NMT still lags behind statistical methods on very low-resource language pairs, while conventional transfer ignores useful source-vocabulary overlap between related languages.
Method
The method maps related source languages to a common alphabet, applies joint BPE segmentation, and transfers model parameters including source embeddings from parent to child.
Results
Transfer helps word-based translation only slightly, but combined with BPE improves over the BPE baseline by 0.8 BLEU for Turkish-English and 4.3 BLEU for Uyghur-English.
Takeaways & Limitations
BPE combined with transfer consistently improves NMT performance in the two studied Turkic child languages by exploiting lexical similarity with a related low-resource language.
Takeaways & Limitations
The approach relies on vocabulary overlap and similar meanings across languages, and its applicability to other language types requires further investigation.
Abstract
from arXiv · showhide
We present a simple method to improve neural translation of a low-resource language pair using parallel data from a related, also low-resource, language pair. The method is based on the transfer method of Zoph et al., but whereas their method ignores any source vocabulary overlap, ours exploits it. First, we split words using Byte Pair Encoding (BPE) to increase vocabulary overlap. Then, we train a model on the first language pair and transfer its parameters, including its source word embeddings, to another model and continue training on the second language pair. Our experiments show that transfer learning helps word-based translation only slightly, but when used on top of a much stronger BPE baseline, it yields larger improvements of up to 4.3 BLEU.
1 Introduction
NMT remains weaker on very low-resource language pairs, motivating transfer from related languages. This paper adapts transfer learning by exploiting source-language vocabulary overlap with BPE, yielding larger gains over a strong BPE baseline.
- NMT still lags behind statistical methods on very low-resource language pairs.
- The paper studies transfer from a related, also low-resource parent language to a child language, where standard transfer does not always work.
- Source word-embedding transfer can preserve useful lexical correspondences when parent and child language lexicons overlap.
- BPE is applied after mapping source languages to a common alphabet to increase shared subwords and vocabulary overlap.
- 0.8 and 4.3 BLEU improvements over BPE are obtained for Turkish and Uyghur, respectively, using Uzbek as the parent language.
2 Background
The background describes parameter transfer between parent and child NMT models, including source embeddings, and introduces BPE and the Turkic-language experimental setting.
- A parent NMT model can initialize a child model, whose parameters are then fine-tuned on low-resource data.
- Because source vocabularies differ, conventional transfer assigns parent source embeddings to child words by position rather than lexical identity.
- Target embeddings are frozen during fine-tuning because the target language is shared by the parent and child models.
- The experiments translate from three related Turkic languages into English.
- Turkic languages are morphologically rich and share similarities in phonology, morphology, syntax, and some word roots.
- BPE iteratively merges frequent character-token pairs into larger symbols, with the number of operations controlling segmentation.
3 Method
The method extends transfer learning by sharing source vocabularies between related languages. It uses orthographic normalization and joint BPE segmentation to preserve or increase lexical overlap.
- The method shares parent and child source vocabularies so transferred embeddings remain aligned for words appearing in both languages.
- Its assumptions are considerable vocabulary overlap and often similar meanings for words shared across the parent and child languages.
- Different orthographies are mapped to a common orthography; Uyghur Arabic script is transliterated into Latin script.
- BPE rules are learned from the combined source and target data of both language pairs, producing one vocabulary for all models.
4 Experiments
The experiments compare word-based and BPE-based transfer systems on Turkish-, Uzbek-, and Uyghur-English data, using case-sensitive BLEU and development-set tuning.
- Experimental setup: The study uses Turkish-, Uzbek-, and Uyghur-English parallel texts from the LORELEI program, with Moses tokenization and script-specific preprocessing.Uyghur word-based models retain Arabic script, while BPE systems use Latin transliteration.
- Systems: Word-based models use vocabulary limits of 45,000 types for Turkish-English and 20,000 for Uyghur-English.Out-of-vocabulary words are replaced with UNK.
- Systems: BPE systems vary the number of merge operations from 5,000 to 60,000 and use the full vocabulary from combined parent and child data.Training also includes unchanged data and a copy replacing words occurring fewer than five times with UNK.
- Training and evaluation: Models use Adadelta, minibatches of 32, dropout 0.2, gradient rescaling above norm 5, beam search, and length normalization.Training stops at the highest tokenized development BLEU, with vocabulary or merge-operation settings selected the same way.
- Training and evaluation: Final evaluation uses case-sensitive BLEU after BPE rejoining, recasing, and detokenization.Table 3 reports case-sensitive test BLEU; Figure 1 reports tokenized development BLEU by vocabulary or subword-type count.
- Systems: The experiments compare BPE and word-based baselines with transfer systems, including variants that fine-tune or freeze target embeddings.The BPE baseline uses randomly initialized weights, while word-based transfer follows Zoph et al.’s vocabulary-sharing-free setup.
5 Results and Analysis
Word-level transfer helps inconsistently, whereas combining transfer with BPE produces larger improvements over the stronger BPE baseline.
- Word-based transfer: Word-based transfer does not consistently improve translation: Turkish-English gains are statistically insignificant, while Uyghur-English benefits depend on whether target embeddings are frozen.For Turkish-English, both methods have p > 0.05; for Uyghur-English, only unfrozen transfer helps somewhat.
- BPE and transfer: BPE models perform much better than word-based counterparts in both language pairs.The BPE baseline provides the stronger platform for evaluating transfer.
- BPE and transfer: BPE-based transfer consistently improves over both the baseline and word-based transfer models, with the authors associating the gain primarily with increased vocabulary overlap.The same pattern appears in best development-set BLEU scores shown in Figure 1.
6 Conclusion
The paper combines transfer learning with BPE to exploit lexical similarity between related low-resource languages and improve NMT performance.
- Conclusion: The method improves low-resource NMT by combining transfer learning with BPE and exploiting lexical similarity between related languages.The conclusion reports consistent improvement for two Turkic child languages.
- Conclusion: The approach appears well suited to agglutinative languages, while its applicability to other language types requires further investigation.This scope boundary is stated as an open question rather than a confirmed limitation across language families.