Source-linked AI summary
Trivial Transfer Learning for Low-Resource Neural Machine Translation
Tom Kocmi, Ondřej Bojar
TL;DR
Low-resource NMT struggles with insufficient parallel data, while earlier transfer-learning methods imposed language-sharing or training constraints. This paper trains a high-resource parent model and continues on low-resource data by replacing the corpus, finding gains across shared and fully unrelated language pairs, including different translation targets.
Problem
NMT struggles when parallel data is insufficient, and prior transfer-learning methods required shared targets, related languages, or special training procedures.
Method
Train a high-resource parent model, then continue training on the low-resource child corpus using a shared subword vocabulary and no training restart.
Results
The method works with shared source or target sides and with fully unrelated language pairs, including pairs with different translation sides.
Takeaways & Limitations
Transfer learning for low-resource NMT can be implemented by changing the training corpus while retaining the existing model and training configuration.
Takeaways & Limitations
The method works well only when the parent has more training data, and alternative vocabulary-balancing strategies were not evaluated.
Abstract
from arXiv · showhide
Transfer learning has been proven as an effective technique for neural machine translation under low-resource conditions. Existing methods require a common target language, language relatedness, or specific training tricks and regimes. We present a simple transfer learning method, where we first train a "parent" model for a high-resource language pair and then continue the training on a lowresource pair only by replacing the training corpus. This "child" model performs significantly better than the baseline trained for lowresource pair only. We are the first to show this for targeting different languages, and we observe the improvements even for unrelated languages with different alphabets.
1 Introduction
NMT performs poorly when parallel data is insufficient, motivating transfer learning for low-resource translation. The paper simplifies prior approaches by removing language-relatedness and shared-target restrictions without special training modifications.
- Insufficient parallel data can make NMT produce fluent outputs unrelated to the source and perform worse than phrase-based translation.
- Prior transfer learning trains a high-resource parent model and reuses its weights for a low-resource child model.
- Earlier methods required a shared target language, language relatedness, subword sharing, transliteration, or training-process modifications.
- The paper removes relatedness and shared-target restrictions and avoids special training-regime or preprocessing modifications.
- The experiments use Transformer models rather than the recurrent architectures used in previous transfer-learning work.
2 Method Description
The method trains a high-resource parent model, then continues training on low-resource child data by changing only the corpus. A shared subword vocabulary connects both language pairs, while alternative vocabulary balancing remains untested.
- The method supports shared source, shared target, or fully unrelated language pairs, including cases where neither translation side overlaps.
- The proposed transfer removes language-relatedness restrictions while retaining shared vocabulary across both models.
- A shared vocabulary of subword units is required across the parent and child language pairs.The vocabulary is learned from concatenated source and target corpora across both pairs.
- Transfer learning switches from a converged parent language-pair model to a child corpus without restarting training.The method keeps the training process continuous while replacing the data source.
- Vocabulary generation balances the language pairs by selecting equal numbers of parent and child sentence pairs.
- The study did not test other vocabulary-balancing strategies, including using only the child corpus or varying the number of selected sentences.
3 Model Description
The experiments use Transformer sequence-to-sequence models trained with fixed optimization and decoding settings. Parent-to-child transfer preserves model parameters and training state, including the shared vocabulary identifiers.
- The experiments use Tensor2Tensor’s Transformer “big single GPU” configuration with a 2300-token batch and 100-wordpiece sentence limit.
- Training uses exponential learning-rate decay, Adam optimization, and 32000 warm-up steps without resetting the learning rate during transfer.
- Decoding uses beam size 8 with length-normalization penalty 1.
- Models train for 1M steps, and the best development-set checkpoint is selected for test evaluation.
- Separate parent and child vocabularies cause an expected performance drop because identical subwords receive different IDs.
4 Datasets
The study compares low-resource Estonian and Slovak with higher-resource Finnish and Czech, using related and unrelated language pairs. Russian, Arabic, French, and Spanish broaden the parent-language experiments.
- Estonian and Slovak are treated as low-resource languages, while Finnish and Czech serve as higher-resource counterparts.
- The datasets span two orders of magnitude in training-data size across the compared language pairs.
- Russian is selected as a parent language because its Cyrillic script is expected to yield little shared-vocabulary overlap with the child languages.
- Arabic, French, and Spanish provide experiments with unrelated languages.
- Training data is filtered by removing sentence pairs shorter than 4 or longer than 75 words on either side.The authors state that this reduction is small and does not change translation-model performance.
- The experiments draw on Europarl, Rapid, WMT, CzEng, News Commentary, Yandex, UN Corpus, and other language-specific resources.
5 Results
Transfer learning improves low-resource NMT when English is shared on either the source or target side, and can also help with unrelated languages and swapped directions. The benefit is strongest when the parent has more training data, including extremely small child datasets.
- English as the Common Language: Up to 2.44 BLEU absolute, sharing the target language improves ETEN over child-only training.
- English as the Common Language: Up to 2.71 BLEU, sharing the source language improves ENET with an ENFI parent.
- English as the Common Language: 3.38 BLEU for ENET with an ENCS parent exceeds the 2.71 BLEU improvement from ENFI, despite Czech and Estonian being unrelated.
- English as the Common Language: Russian-to-Estonian transfer improves both directions without transliteration despite no shared vocabulary beyond numbers and punctuation.
- English as the Common Language: The method generally works only when the parent has more training data; low-resource parents can fail to improve or significantly reduce performance for better-resourced children.
- Direction Swap in Parent and Child: Transfer gains are mostly significant for unrelated languages, ranging from +0.49–+0.78 BLEU; Arabic-Russian is the only non-significant case.
6 Analysis
The analysis examines vocabulary overlap and output changes as possible sources of transfer-learning gains. Improvements persist across multiple evaluation scores and are not explained solely by longer outputs or simple shared token types.
- 6.1 Vocabulary Overlap: 8.89% of subwords are seen in all three languages in the ET–EN–RU vocabulary analysis.English and Estonian each have 20–30% language-unique subwords, while Russian shares 0–1.39% with each other language.
- 6.1 Vocabulary Overlap: 41.03% of subwords fall into categories where parent training can help the child model.These include English subwords and tokens common to Estonian and Russian; their embeddings receive parent training.
- 6.1 Vocabulary Overlap: 6.2% of subwords were available in the parent for AR-RU-ET-EN, yet the child still gained BLEU.This case suggests that substantial vocabulary knowledge from the parent is not required for an observed gain.
- 6 Analysis: The improvements remain apparent across several cased automatic scores, not only uncased BLEU.The analysis compares the ENET baseline with children derived from ENCS and ENRU parents.
- 6.2 Output Analysis: Improved outputs are considerably longer, but gains also appear in n-gram precisions rather than arising solely from reduced brevity penalty.Table 9 reports candidate length, BLEU n-gram precisions, and brevity penalty.
- 6.2 Output Analysis: About 44+20% of improved-output tokens are unchanged from the baseline, with 44% confirmed by the reference.The 44% “rb” tokens appear in both the improved output and the reference.
- 6.2 Output Analysis: For both parent setups, about 9–9.7% of tokens are present in improved outputs and references but absent from the baseline.These differing tokens are regular Estonian words, and gains remain identical after removing punctuation, numbers, and named entities.
7 Related Work
Related work addresses low-resource translation through multilingual transfer, shared or related languages, unsupervised learning, transliteration, monolingual-data augmentation, and curriculum learning.
- Multilingual Systems: Multi-way multilingual systems reduce parameters across language pairs but require special scheduling to prevent catastrophic forgetting.Without keeping earlier language pairs active, previously learned translation abilities can be lost.
- Multilingual Systems: Another multilingual approach trains all translation pairs jointly and uses a special source-side token to indicate the target language.The model can translate language pairs that were never directly seen together.
- Unsupervised and Monolingual Methods: Unsupervised translation combines monolingual autoencoder training with translation on data produced by the previous system iteration.This approach targets settings without parallel data.
- Unsupervised and Monolingual Methods: Transliteration can use closely related high-resource training data when no parallel data are available.The cited approach specifically addresses closely related languages.
- Unsupervised and Monolingual Methods: Back-translation and copying target monolingual data to the source can improve translation quality in low-data conditions.Both methods augment training using target-side monolingual material.
- Curriculum Learning: Curriculum learning orders training examples from foreign out-of-domain data toward in-domain examples.This provides a related training-data scheduling strategy.
8 Conclusion
The paper presents a simple parent-to-child transfer method for low-resource NMT and finds gains across shared and unrelated language pairs. The source of these gains remains unresolved, but parent-corpus size appears more important than vocabulary overlap.
- 8 Conclusion: The method trains on a high-resource parent pair, then continues training on a low-resource child pair without resetting training.It works when the language pairs share a source or target side and when they share neither.
- 8 Conclusion: The method also produces gains for totally unrelated language pairs, although the gains are not always significant.This extends the observed transfer effect beyond shared-language settings.
- 8 Conclusion: Starting training in the opposite direction and later swapping to the main direction is proposed as a trick for low-resource languages.This is presented as an additional technique in the conclusion.
- 8 Conclusion: The gains’ detailed causes remain unexplained, while observations indicate parent-corpus size matters more than vocabulary overlap.The conclusion frames this as an indication rather than a definitive mechanism.