Source-linked AI summary
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan
TL;DR
Multilingual translation needed scalable models that could use multilingual pretraining while supporting low-resource languages and additional languages. The paper introduces multilingual finetuning of pretrained models, extends mBART to 50 languages, and releases ML50; on average, it improves 1 BLEU over the strongest baselines.
Problem
Specialized language models do not scale, while low-resource languages often lack abundant bitext and pretrained models are limited to their original language set.
Method
The paper finetunes a multilingual pretrained model simultaneously on multiple translation directions using multilingual bitext.
Results
1 BLEU improvement over the strongest baseline was achieved on average across Many-to-English and English-to-Many settings.
Takeaways & Limitations
Pretrained multilingual models can be extended to additional languages without performance loss on the original languages, supporting mBART50 with 50 languages.
Takeaways & Limitations
One-to-Many and Many-to-Many decoding must generate 50 languages, and lower-resource languages have limited monolingual data.
Abstract
from arXiv · showhide
Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages. Previous work in multilingual pretraining has demonstrated that machine translation systems can be created by finetuning on bitext. In this work, we show that multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one direction, a pretrained model is finetuned on many directions at the same time. Compared to multilingual models trained from scratch, starting from pretrained models incorporates the benefits of large quantities of unlabeled monolingual data, which is particularly important for low resource languages where bitext is not available. We demonstrate that pretrained models can be extended to incorporate additional languages without loss of performance. We double the number of languages in mBART to support multilingual machine translation models of 50 languages. Finally, we create the ML50 benchmark, covering low, mid, and high resource languages, to facilitate reproducible research by standardizing training and evaluation data. On ML50, we demonstrate that multilingual finetuning improves on average 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while improving 9.3 BLEU on average over bilingual baselines from scratch.
1 Introduction
The paper proposes multilingual finetuning of pretrained models to build translation systems across many directions, addressing limited scalability and low-resource data. It extends mBART to 50 languages and introduces ML50 for standardized multilingual translation evaluation.
- Motivation: Specialized models for each language are difficult to scale, especially where labeled data and bitext are scarce.Monolingual data remains available for some low-resource languages through sources such as Wikipedia and Commoncrawl.
- Motivation: Multilingual finetuning adapts a pretrained multilingual model across many translation directions instead of using only bilingual finetuning.This approach is intended to leverage the multilingual nature of pretraining more fully.
- Contributions: mBART was extended from 25 to 50 supported languages without training from scratch or performance loss on the original languages.The resulting mBART50 model was released for community use.
- Results: 3.6 BLEU improvement over bilingual finetuning and 2.6 BLEU improvement over multilingual models trained from scratch were achieved in the Many-to-English setting.
- Results: 1 BLEU improvement over the strongest baseline was achieved on average across Many-to-English and English-to-Many settings.
- Contributions: The ML50 benchmark covers high-, mid-, and low-resource languages and provides publicly available multilingual translation training and evaluation data.Its dataset consists of 230M bitext.
2 Related work
Related work established multilingual pretraining and multilingual translation systems, but this paper extends pretrained multilingual models through multilingual rather than bilingual finetuning.
- Multilingual pretraining: Multilingual pretrained models improved finetuning for machine translation between two languages.
- Positioning: This work extends mBART by allowing finetuning in multilingual settings rather than only bilingual settings.
- Multilingual translation: Earlier multilingual translation systems trained multiple directions jointly from scratch and improved low-resource translation, including enabling zero-shot translation.
- Multilingual translation: Prior work indicated that sufficiently large models are needed to leverage massive multilingual corpora fully.
- Positioning: A concurrent approach jointly trained multilingual systems with monolingual datasets, whereas this paper focuses on finetuning translation systems from a pretrained model.
3 Multilingual Translation from Denoising Pretraining
The method uses multilingual denoising pretraining as the starting point and multilingual finetuning on bitext to create one model for many translation directions. It supports several direction configurations and temperature sampling to reduce imbalance across language pairs.
- Denoising pretraining: mBART is a sequence-to-sequence generative pretraining scheme that uses multilingual monolingual data and denoising autoencoding.The model predicts original text from a corrupted version.
- Denoising pretraining: The pretraining noise function uses random span masking and order permutation.
- Multilingual finetuning: Multilingual finetuning adapts the pretrained model into one system capable of translating many languages to many other languages.The authors associate this with efficiency and storage-maintenance benefits.
- Multilingual finetuning: Multilingual finetuning collects bitext from multiple language directions and initializes a sequence-to-sequence model with pretrained mBART.Source and target language tokens are added before finetuning on the combined multilingual bitext.
- Model variants: The model is evaluated in Many-to-one, one-to-Many, and Many-to-Many configurations.Many-to-one decodes to English, one-to-Many decodes from English, and Many-to-Many uses a pivot language.
- Training balance: Temperature upsampling increases sampling of lower-resource language pairs so high-resource languages do not dominate training.
4 Results from Multilingual Finetuning on 25 Languages
On 25 languages, multilingual finetuning of mBART outperformed key baselines most clearly for translation into English, while gains were smaller or negative in directions from English.
- Performance comparison: 1.0 BLEU improvement over the strongest baseline was achieved by multilingual finetuning in the Many-to-one setting.The strongest baseline was multilingual training from scratch.
- Performance comparison: 7.9 BLEU points stronger than bilingual training from scratch, multilingual finetuning was in the Many-to-one setting.It was also 1.0 BLEU stronger than multilingual training from scratch.
- Performance comparison: 3 BLEU points stronger than bilingual training from scratch, multilingual finetuning was in the one-to-Many setting but 0.6 BLEU lower than bilingual finetuning.The authors hypothesize that decoding into many languages makes this setting more difficult.
- Performance comparison: 3.98 BLEU points stronger than bilingual training from scratch, multilingual finetuning was in the Many-to-Many setting for directions involving English.The authors again hypothesize that decoding into many languages contributes to lower gains.
- Performance by resource level: 18.0 BLEU points improvement over bilingual training from scratch occurred for Many-to-one directions with 7K-30K available bitext, versus 4.4 BLEU points above 10M bitext.The improvement from multilingual training became more significant as available training bitext decreased.
5 Results from Multilingual Finetuning on 50 Languages
The authors extend mBART from 25 to 50 languages and introduce ML50, then evaluate multilingual finetuning across resource levels and translation settings.
- ML50 benchmark: ML50 standardizes training and evaluation across 50 languages spanning extremely low- to high-resource settings.The benchmark includes publicly available training and evaluation sets and groups languages by available bitext.
- ML50 results: 2.5 BLEU points improvement over the strongest baseline was obtained by multilingual finetuning in Many-to-one translation across all ML50 languages.For 4K-10K training pairs, the improvement was 4.8 BLEU points; for 10K-100K pairs, it was over 4 BLEU.
- ML50 results: 0.8 BLEU improvement over the strongest baseline was achieved in Many-to-Many translation to English, while from-English performance was similar to multilingual training from scratch.In one-to-Many translation, methods were similar, with roughly 5.7 to 7 BLEU improvement over bilingual baselines.
- Comparison to bilingual finetuning: 3.6 average BLEU improvement over bilingual finetuning was observed for translation into English, compared with a −0.47 BLEU average difference from English.The figure compares multilingual and bilingual finetuning across 50-language translation directions.
6 Discussion
Multilingual finetuning is strongest overall because it substantially improves Many-to-one translation, while one-to-Many performance remains similar across approaches. Extending mBART from 25 to 50 languages preserves performance on the original languages.
- 6 Discussion: Many-to-one gains arise partly because the model sees English target-side data from 50 language pairs.
- 6 Discussion: One-to-Many and Many-to-Many translation remain difficult because the model must decode into 50 languages with uneven monolingual-data availability.
- 6 Discussion: Multilingual finetuning outperforms bilingual training from scratch, bilingual finetuning, and multilingual training from scratch when averaged across Many-to-one and one-to-Many directions.
- 6.2 Comparison of mBART50 on 25 Languages: mBART can be extended from 25 to 50 languages without performance reduction on the original 25 languages.
7 Conclusion
The paper turns pretrained mBART into multilingual translation systems through multilingual finetuning and extends it to 50 languages. It reports stronger overall performance than the assessed baselines and releases mBART50 and the ML50 benchmark.
- 7 Conclusion: Multilingual translation models can be created from pretrained models such as mBART through multilingual finetuning.
- 7 Conclusion: mBART is extended to 50 languages without loss of performance on the original languages.
- 7 Conclusion: The authors release mBART50 and the ML50 benchmark for multilingual translation in 50 languages.
- 7 Conclusion: Multilingual finetuning achieves over 2 BLEU points of improvement in the Many-to-one setting and outperforms all baselines overall.
A Appendices
The appendices provide dataset statistics and comparative result tables for ML50, including bilingual, multilingual, and multilingual-finetuning baselines.
- A Appendices: Table 6 lists per-language training-data size, training and evaluation data sources, and evaluation-data size after filtering.
- A Appendices: Table 6 notes that some available data are missing for a few language pairs because of human error, with a corrected release planned.
- A Appendices: Table 7 reports bilingual and bilingual-finetuning baselines over 50 languages.
- A Appendices: Table 8 reports multilingual baselines over 50 languages.
- A Appendices: Table 9 reports multilingual-finetuning results over 50 languages.