Source-linked AI summary
Massively Multilingual Neural Machine Translation
Roee Aharoni, Melvin Johnson, Orhan Firat
TL;DR
The paper asks how far a single NMT model can scale across many language pairs while balancing transfer benefits against capacity bottlenecks. It evaluates massively multilingual English-centric many-to-many models across low- and high-resource settings, finding improved translation quality over bilingual baselines and stronger zero-shot generalization, while identifying capacity limits and possible harms for some language pairs.
Problem
The paper investigates how well one NMT model can scale to a very large number of language pairs, given potential transfer benefits and model bottlenecks.
Method
The authors train and evaluate English-centric massively multilingual Transformer models across low- and high-resource datasets, varying languages, directions, and model capacity.
Results
Models supporting 102 languages and 204 directions improve translation quality by more than 2 BLEU over comparable bilingual baselines across 10 diverse language pairs, while 59-language models also outperform bilingual models in low-resource TED Talks experiments.
Takeaways & Limitations
Massively multilingual NMT can scale to large language sets, improve translation over bilingual baselines, and generalize better to zero-shot settings.
Takeaways & Limitations
The experiments indicate capacity bottlenecks even with very large models, and joint training can harm performance for some language pairs.
Abstract
from arXiv · showhide
Multilingual neural machine translation (NMT) enables training a single model that supports translation from multiple source languages into multiple target languages. In this paper, we push the limits of multilingual NMT in terms of number of languages being used. We perform extensive experiments in training massively multilingual NMT models, translating up to 102 languages to and from English within a single model. We explore different setups for training such models and analyze the trade-offs between translation quality and various modeling decisions. We report results on the publicly available TED talks multilingual corpus where we show that massively multilingual many-to-many models are effective in low resource settings, outperforming the previous state-of-the-art while supporting up to 59 languages. Our experiments on a large-scale dataset with 102 languages to and from English and up to one million examples per direction also show promising results, surpassing strong bilingual baselines and encouraging future work on massively multilingual NMT.
1 Introduction
Multilingual NMT trains one model for multiple language pairs, offering efficiency, transfer learning, and zero-shot translation. This work studies how massively multilingual models scale in languages, directions, capacity, and resource settings.
- Multilingual NMT trains a single model capable of translating between multiple language pairs.Such models can simplify deployment by reducing the number of required models and parameters.
- Transfer learning may improve low-resource translation when language pairs are trained alongside high-resource pairs.An extreme form is zero-shot translation between language pairs unseen during training.
- Previous multilingual NMT work typically supported up to 7 languages and 20 trained directions, while one exception used 58-to-English models.The 58-language experiments were limited to low-resource many-to-one translation, with up to 214k examples per pair and evaluation on four directions.
- The paper trains English-centric many-to-many Transformer models supporting up to 102 languages and one million examples per language pair.The setting uses language pairs containing English on either the source or target side, reflecting the availability of English parallel data.
- Experiments vary model capacity, trained directions, and resource settings using TED talks and a large-scale dataset with 102 languages aligned to and from English.The study reports low-resource results with up to 58 languages and high-resource models trained on 204 translation directions.
2 Low-Resource Setting: 59 Languages
The TED Talks experiments test English-centric multilingual NMT across 59 languages and 116 directions, comparing many-to-many, one-to-many, many-to-one, and bilingual systems under low- and higher-resource conditions. Many-to-many models improve X→En performance in low-resource settings, while one-to-many models are stronger for En→X, and the comparisons were conducted without separate hyperparameter tuning.
- The TED Talks corpus contains parallel data for 59 languages, with English-centric training covering 116 translation directions and 3.3k–214k training examples per pair.
- The experiments compare many-to-many, one-to-many, and many-to-one models against bilingual baselines using identical Transformer-based training conditions.The many-to-many model uses all 116 directions, while the one-to-many and many-to-one models each cover 58 directions.
- 1.82 BLEU improvement over Neubig and Hu’s fine-tuned many-to-one models and 2.44 BLEU over the authors’ many-to-one model were achieved by many-to-many systems on four low-resource X→En pairs.The many-to-many model used the same X→En data, architecture, and capacity while supporting 58 additional English→X tasks.
- Bilingual baselines outperform many-to-one models only for higher-resource language pairs, whereas many-to-one models benefit from the low-resource setting.The authors relate this pattern to the greater amount of training data available to higher-resource bilingual baselines.
- Many-to-many models outperform many-to-one and bilingual models when translating into English in the low-resource setting, with additional target languages potentially reducing overfitting.Training curves show many-to-one development BLEU degrading while many-to-many development BLEU continues improving, alongside a large training–development gap for many-to-one models.
- The comparisons used identical training conditions rather than extensive setting-specific hyperparameter tuning, which may affect the relative results.The authors note that batch diversity differs substantially across configurations and may require tailored batch sizes, learning rates, or dropout rates.
3 High-Resource Setting: 103 Languages
The high-resource experiments scale English-centric NMT to 103 languages and 204 directions, using up to one million examples per language pair. Massively multilingual models outperform bilingual baselines on average, while model configuration affects direction-specific performance.
- 103 languages and 204 translation directions are trained simultaneously, with up to one million examples per language pair.
- The evaluation covers 10 typologically diverse languages in both directions with English.
- For translation into English, the many-to-one model generally outperforms the many-to-many model when data is abundant and not multi-way-parallel.The paper attributes the difference to the absence of many-to-one overfitting in this setting.
- German-to-English is an exception: the many-to-one model scores 2 BLEU points below the many-to-many model, with unstable development performance.The reported comparison identifies German-to-English as an outlier among the evaluated directions.
- Both massively multilingual models outperform bilingual baselines on average for translation out of English, while one-to-many performs better than many-to-many.The paper relates this pattern to the smaller number of tasks and lack of target-side English bias in one-to-many models.
4 Analysis
The analysis studies how the number of jointly trained languages affects supervised and zero-shot accuracy under controlled settings. More languages can reduce supervised accuracy at fixed capacity, while medium-to-large multilingual models show stronger zero-shot generalization, with capacity limits remaining.
- 4.1 Multilinguality & Supervised Performance: The analysis varies the number of languages while keeping model capacity and training configuration fixed to measure the accuracy trade-off.
- 4.1 Multilinguality & Supervised Performance: The supervised experiment uses nested English-centric subsets containing 5, 25, 50, and 75 languages to-and-from English.
- 4.1 Multilinguality & Supervised Performance: In most cases, the 5-to-5 model achieves the best supervised results, showing a trade-off between language count and accuracy at fixed capacity.Models with more languages see fewer examples per language in each batch, which may make their gaps shrink with additional updates.
- 4.2 Multilinguality & Zero-Shot Performance: Zero-shot evaluation compares Arabic↔French and Ukrainian↔Russian, representing distant and similar language pairs.The study treats zero-shot accuracy as a measure of model generalization.
- 4.2 Multilinguality & Zero-Shot Performance: 50-to-50, 75-to-75, and 103-to-103 models show better zero-shot performance than the 5-to-5 and 25-to-25 models.Figure 2 illustrates the stronger generalization of the larger multilingual models in the Ukrainian-to-Russian setting.
- 4.2 Multilinguality & Zero-Shot Performance: The 50-to-50 model offers the best balance between capacity and generalization, while the setting may already be hitting a capacity barrier.The paper notes that even models with more than 473M trained parameters may require still larger capacity.
5 Related Work
Prior multilingual NMT work explored parameter sharing, zero-resource translation, and multilingual transfer, but generally used far fewer languages or directions. Related massively multilingual studies reached larger language counts in some cases without evaluating produced translations or focused on adaptation.
- Early multilingual NMT extended one-to-many systems to four target languages and many-to-many systems to up to six languages using language-specific components and shared mechanisms.
- Subsequent work investigated alternative parameter-sharing strategies, including shared components, interlingua layers, contextual parameter generation, and language experts.
- Most earlier multilingual NMT studies used up to 7 languages or 20 trained directions in one model, whereas this work targets substantially larger scales.
- Neubig and Hu trained multilingual models on the 59-language TED Talks corpus for rapid adaptation to new languages, rather than focusing only on direct massively multilingual translation.
- Other studies trained English-centric or many-to-English systems on Bible translations with 927 or 1017 languages, but did not evaluate the produced translations in the cited work.
6 Conclusions and Future Work
The paper demonstrates that massively multilingual NMT can scale to over 100 languages and improve translation quality over comparable bilingual systems. It also identifies capacity bottlenecks, language-specific interference, and stronger zero-shot generalization as important directions for future work.
- NMT scales to 102 languages to-and-from English with 204 trained directions and up to one million examples per direction.
- On 10 diverse language pairs, these models improve translation quality over similar single-pair baselines by more than 2 BLEU in each direction.
- On the low-resource TED Talks corpus, massively multilingual models achieve similar gains while supporting 59 languages and 116 trained directions.
- The analysis finds capacity bottlenecks as language count increases and shows that massively multilingual models can generalize better in zero-shot settings.
- Future work includes semi-supervised learning, reducing performance degradation with more languages, multilingual transfer learning, and improving zero-shot performance.
A Supplementary Material
The supplementary material lists language-pair coverage for the TED talks and in-house datasets. The datasets pair languages with English, with the in-house data providing up to one million examples for most languages.
- The TED talks dataset contains 58 languages paired with English, with train-set sizes reported for each pair.
- Afrikaans and Laothian are among the languages listed in the supplementary language-pair materials.
- The in-house dataset contains 102 languages paired with English; languages marked with * have fewer than 1M examples, while the others have exactly 1M.