Source-linked AI summary
Beyond English-Centric Multilingual Machine Translation
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin
TL;DR
English-Centric multilingual translation relies on data involving English and performs worse for non-English directions. This paper builds M2M-100 with directly mined Many-to-Many data for 100 languages and increased dense and sparse capacity, improving non-English translation by over 10 BLEU on average while remaining competitive with bilingual WMT models.
Problem
English-Centric datasets and models provide limited direct coverage of non-English translation directions and empirically yield lower performance on them.
Method
The authors mine a 7.5B-sentence Many-to-Many dataset for 100 languages using bridge-language groupings, then scale Transformers with dense and language-specific sparse parameters.
Results
M2M-100 improves over an English-Centric baseline by over 10 BLEU on average for direct non-English directions and is competitive with bilingual WMT models.
Takeaways & Limitations
Direct Many-to-Many translation across 100 languages can outperform English-Centric multilingual systems on non-English directions without pivoting through English.
Takeaways & Limitations
Performance remains in need of substantial improvement for many very low-resource languages because limited monolingual resources constrain the quantity and quality of mined data.
Abstract
from arXiv · showhide
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. However, much of this work is English-Centric by training only on data which was translated from or to English. While this is supported by large sources of training data, it does not reflect translation needs worldwide. In this work, we create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Then, we explore how to effectively increase model capacity through a combination of dense scaling and language-specific sparse parameters to create high quality models. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT. We open-source our scripts so that others may reproduce the data, evaluation, and final M2M-100 model.
1. Introduction
Prior multilingual translation systems were largely English-Centric, limiting direct non-English translation. This work builds a Many-to-Many dataset and scales a multilingual model to translate directly among 100 languages.
- English-Centric datasets train primarily on directions involving English, which does not reflect worldwide translation needs and lowers non-English translation performance.
- 7.5B training sentences across 100 languages provide direct supervised data for thousands of translation directions.
- The model combines dense scaling with language-specific sparse parameters to address the capacity demands of Many-to-Many training.
- M2M-100 directly translates between 100 languages without pivoting through English while remaining competitive with bilingual models on benchmarks including WMT.
2. Preliminaries
The paper reviews Transformer-based multilingual translation, from tokenization and language conditioning to encoder-decoder computation and large-scale training. It also describes sampling and sharding choices for supporting diverse languages and resources.
- Multilingual translation uses a single model to translate between language pairs while sharing computation and information across languages.
- Segmentation with SentencePiece: SentencePiece supplies multilingual subword units and is suited to languages without explicit word segmentation.
- Creating a Multilingual Dictionary: Temperature sampling with T = 5 and added monolingual data reduce underrepresentation of low-resource languages in the multilingual dictionary.
- Transformer architecture: The encoder maps source tokens and source language information to embeddings, while the decoder autoregressively generates target tokens conditioned on the target language.
- Training: The baseline Transformer has 12 encoder and 12 decoder layers, 8192-dimensional feed-forward networks, 1024-dimensional embeddings, and 1.2B parameters.
3. Building a Many-to-Many Parallel Dataset for 100 Languages
The dataset covers 100 selected languages and thousands of evaluation directions using mined parallel data, backtranslation, and publicly available benchmarks. Language selection emphasizes diversity, evaluability, and mining resources.
- The Many-to-Many dataset provides training data for thousands of directions among 100 languages through mining and backtranslation.
- Language Selection: The language set prioritizes globally spoken languages, geographic and family diversity, script and resource diversity, public evaluation data, and available monolingual data.
- Evaluation Benchmarks: Evaluation combines WMT, WAT, IWSLT, FLORES, TED, Autshumato, and Tatoeba datasets spanning news, talks, Wikipedia, government, and mixed domains.
- Evaluation: The study evaluates translation quality with BLEU after language-specific tokenization and normalization procedures.
3.2 Covering the Language Matrix by Mining Relevant Parallel Data
The paper constructs a scalable Many-to-Many mining strategy that combines language-group and bridge-language selection instead of exhaustively mining every language pair. This produces broad non-English coverage while retaining substantial data volume and performance.
- Mining Strategy: Mining all language pairs is computationally prohibitive, motivating sparse selection beyond the 99 English-linked directions used by prior work.The paper notes that random selection ignores linguistic relationships and real-world language use.
- Mining Strategy: The strategy groups 100 languages into 14 linguistic, geographic, or cultural groupings and mines all pairs within each grouping.Grouping languages supports direct translation among related languages and supplies language-specific training data.
- Mining Strategy: It adds 1–3 resource-rich bridge languages per grouping, mines bridges across groupings, and mines every language against English.This connects otherwise separate groupings without requiring exhaustive pairwise mining.
- Data Coverage: The final corpus contains 7.5B parallel sentences across 2200 directions, with Many-to-Many mining yielding 5–10 times more parallel data than English-Centric mining.The additional data is particularly beneficial for mid- and low-resource languages.
- Results: At fixed sparsity, bridge-language mining outperforms random alternatives because it prioritizes high-resource and related language pairs that produce more bitext and broader direction coverage.The strategy avoids randomly selecting pairs that generate little data.
- Results: Increasing mined data helps, but fully mining the language matrix adds limited quality because later pairs contribute little additional bitext.The authors describe this as diminishing quality improvements from increasingly low-yield pairs.
3.3 Augmenting Bitext Data with Backtranslation
The paper uses backtranslation to augment supervised bitext for initially weak translation directions. Applied to selected low-performing pairs, this augmentation almost always improves performance.
- Backtranslation: Backtranslation creates synthetic bitext by translating monolingual target-language sentences backward and adding the resulting translations to training data.This supplements aligned parallel data with synthetic sentence pairs.
- Pair Selection: The authors select 100 directions whose 1.2B-parameter multilingual model initially scores between 2 and 10 BLEU.The selection targets directions with low initial translation quality because backtranslation is computationally intensive.
- Training: Synthetic translations are added without upsampling and marked with a special encoder-side BT token.The token indicates to the model that the added translations are synthetic.
- Data Generation: For each selected target language, the method samples 50 million monolingual CommonCrawl sentences and generates synthetic translations with the 1.2B multilingual model.Generation uses beam search with beam size 5 and fixed hyperparameters across directions.
- Results: Backtranslation almost always improves performance across the explored directions, including directions spanning different original BLEU scores.The authors therefore focus BT on directions initially scoring between 2 and 10 BLEU.
3.4 Balancing Languages in a Many-to-Many Setting
Many-to-Many language distributions are interdependent, so the paper balances source and target languages jointly while preserving the absence of unsupported language pairs.
- Sequentially sampling source and target languages cannot realize the desired target distribution when languages have restricted pairing patterns.Pair availability and varying bitext sizes make independent or sequential sampling inadequate.
- The method estimates a square pair-probability matrix whose row and column sums match the target probability of each language.The vector p contains language probabilities, while Q contains pair probabilities.
- Sinkhorn-Knopp solves the matrix-scaling problem while preserving zero probabilities for language pairs without bitext.The matrix is computed before training, with temperature T set to 5.
- The proposed balancing strategy yields a constant improvement of 0.5 BLEU over temperature sampling.
4. Many-to-Many Compared to English Centric
Many-to-Many training matches English-Centric performance on directions involving English but substantially improves direct translation between non-English languages, including zero-shot directions.
- Main Result: Many-to-Many achieves 31.2 BLEU to English, 24.1 from English, and 15.9 on non-English directions.The evaluation aggregates 150 directions for each English-involving category and 2500 non-English directions.
- Main Result: Many-to-Many matches English-Centric performance on directions involving English while significantly outperforming it on non-English directions.
- Main Result: 10.2 BLEU is the Many-to-Many gain over direct English-Centric translation on non-English pairs, while the gain over English-Centric pivoting is 5.5 BLEU.
- Understanding the Source of Improvement: Nearly 11 BLEU separates Many-to-Many from direct English-Centric translation on non-English pairs without bitext, while pivoting retains a gain above 6 BLEU.
- Understanding the Source of Improvement: With equal data exposure, English-Centric performs better to English, but Many-to-Many outperforms it from English and on non-English directions.The authors attribute the English-Centric advantage to learning only one output language, while Many-to-Many benefits from broader data.
- Understanding the Source of Improvement: Improvements correlate with bilingual and target-language data, while related-language transfer enables Belarusian to reach 12.7 BLEU versus 3.2 BLEU for a bilingual model.The Belarusian result is linked to knowledge transfer from Russian.
5. Components for Scaling Multilingual Translation Models
Scaling the multilingual model favors wider Transformers and language-specific parameters, which add capacity for language groups without proportionally increasing computation.
- 5.1 Scaling Dense Models: Wider models scale better than deeper models in BLEU and words per second across 1B, 2B, and 10B models.The comparison evaluates 38 benchmark directions across 80 languages.
- 5.1 Scaling Dense Models: Increasing dense capacity from 418M to 12B parameters improves performance even on low-resource pairs, but gains grow roughly logarithmically.Scaling from 1.2B to 12B yields +1.5 BLEU, while runtime and memory become increasingly prohibitive.
- 5.2 Language-Specific Parameters: Language-specific layers split parameters by language or language group so each direction accesses only a subset of specialized capacity.The design targets capacity scaling without significantly affecting training and inference time.
- 5.2 Language-Specific Parameters: Languages are grouped using training-data frequency and vocabulary overlap, producing 46 groups including 28 high-resource language-specific groups.Remaining languages form 18 additional groups based primarily on vocabulary similarity and balanced data volume.
- 5.2 Language-Specific Parameters: Language-specific layers outperform similarly sized dense baselines, particularly for mid- and high-resource languages.The paper also reports that decoder specialization is generally more effective in the tested high-resource setting.
- 5.2 Language-Specific Parameters: Adding language-specific decoder layers to a pretrained 12B Transformer improves WMT results after adding 3.4B parameters and freezing the remaining network.The resulting model contains 15.4B parameters and is trained for 20K updates.
6. Bringing it all Together
The evaluation shows that M2M-100 improves non-English translation substantially, remains competitive with bilingual WMT systems, and receives strong human semantic-accuracy ratings. The discussion also identifies training-data cleanliness, domain mismatch, and very low-resource languages as continuing challenges.
- Evaluation overview: The section evaluates M2M-100 on multilingual benchmarks, WMT directions, human judgments, and non-English translation pairs.The comparisons include published multilingual and bilingual systems, WMT single-model results, and human evaluations.
- Non-English directions: Across non-English directions, M2M-100 improves over English-Centric models by more than 7 BLEU on average.The comparison covers official and regional-language directions that do not involve English.
- Bilingual comparisons: +0.7 BLEU improvement is reported in one comparison with tuned bilingual models.The evaluation tunes beam size, length penalty, and checkpoint averaging following the comparison setting.
- Bilingual comparisons: 2 BLEU points is the average gain over hundreds of individual bilingual models in the CCMatrix comparison.M2M-100 uses one Many-to-Many model, while the baselines train a separate model for each direction.
- Non-English directions: More than 4 BLEU improvement is reported on the OPUS100 non-English directions.The evaluation uses beam size 5, length penalty 1, and the best checkpoint.
- Human evaluation: Human evaluations find most semantic-accuracy scores between 8.5 and 9.5, while lower-resource directions remain reasonable and French-to-Wolof is most challenging.Hindi-to-Tamil and Wolof-to-French score around 7–8; insufficient target-side Wolof data is suggested as a likely difficulty.
- Discussion: Training quality remains constrained by noisy or heterogeneous mined data, domain mismatch, and limited resources for very low-resource languages.The paper names examples including Xhosa, Zulu, Catalan, Basque, Iloko, and Cebuano.
7. Conclusion
The conclusion presents M2M-100 as a Many-to-Many model covering 9,900 directions across 100 languages, supported by mined data and dense-plus-sparse scaling. It reports gains over English-Centric multilingual systems, competitiveness with bilingual WMT models, and strong human-rated fluency and semantic accuracy.
- Contribution: M2M-100 translates between 9,900 directions of 100 languages.Its dataset is mined from CommonCrawl using language groupings to avoid mining every possible direction.
- Method: The model combines dense parameter scaling with language-specific sparse parameters trained through a random re-routing scheme.The scaling approach addresses the increased capacity demands of the Many-to-Many dataset.
- Results: More than 10 BLEU average improvement is achieved over an English-Centric baseline for direct non-English translation.The result compares M2M-100 with multilingual models trained only when either the source or target language is English.
- Results: M2M-100 is competitive with bilingual WMT models and improves over existing publicly available multilingual translation systems.Human judges also report fluent translations with high semantic accuracy.
Appendix A. Additional Information about Data
The appendix provides additional information about dictionary coverage across the model’s 100 languages.
- Dictionary coverage: Figure 13 displays dictionary coverage for each of the 100 languages.The figure organizes coverage by language.
Appendix B. Model Architectures
The appendix documents the dense-model configurations considered during scaling and presents the architecture of the wide and deep models.
- Dense models: Table 14 lists the model configurations considered when scaling dense models.These configurations support the experiments on dense capacity scaling.
- Wide and deep models: Figure 14 presents the architecture of the wide and deep models.The figure concerns the model architectures used in the scaling study.
Appendix C. Exploiting Multilinguality at Inference Time with Multi-source Self-Ensembles
The paper explores exploiting multilinguality during inference through self-ensembles, which combine translations generated by one multilingual model from the same source sentence in different languages. It evaluates pivoting and self-ensembling on randomly sampled zero-shot directions.
- Self-ensembling applies the multilingual model to the same source sentence in different languages instead of training multiple models.This is proposed as an alternative to standard model ensembles, which substantially increase training compute requirements.
- Table 12 reports average test BLEU for Multi-Source Self-Ensemble against various baselines across 100 randomly sampled language pairs.
- For Galician-to-English translation, the method combines Spanish-to-English and Galician-to-English predictions by averaging token log-probabilities.The additional source is obtained by translating the input into an intermediary language.
- The evaluation compares pivoting and self-ensembling on zero-shot directions, where improved accuracy may be beneficial.Results are reported on 100 randomly sampled zero-shot translation directions with at least 1000 validation and test examples each.
- For each translation direction, the intermediary language is selected using the highest validation BLEU for the corresponding method.The evaluation also tunes a weight to balance the two languages.