Source-linked AI summary
mT5: A massively multilingual pre-trained text-to-text transformer
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel
TL;DR
English-centric language-model pretraining limits multilingual NLP, while generative models can produce wrong-language zero-shot outputs. This paper introduces mT5 and mC4 across 101 languages, achieving state-of-the-art results on many multilingual benchmarks and exceeding SOTA on classification and QA tasks.
Problem
English-only pretraining limits language-model use for the roughly 80% of people who do not speak English, while generative models can produce wrong-language zero-shot predictions.
Method
The paper develops mT5, a multilingual T5 variant pretrained on mC4, a Common Crawl-based dataset containing natural text in 101 languages.
Results
mT5-XXL exceeds state-of-the-art on all classification and QA tasks and approaches SOTA on NER, scoring 69.2 versus 70.1.
Takeaways & Limitations
Scaling a simple multilingual pretraining recipe can provide a viable alternative to methods relying on language-model filtering, parallel data, or intermediate tasks.
Takeaways & Limitations
Constraining outputs to prevent accidental translation would depart from the general text-to-text interface and does not extend easily to open-ended zero-shot generation.
Abstract
from arXiv · showhide
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.
1 Introduction
mT5 addresses the limitations of English-centric pretraining by introducing a massively multilingual variant of T5. It preserves T5’s general-purpose text-to-text design and scale while providing pretrained models and code for community use.
- Motivation: Many language models were pretrained solely on English text, limiting their usefulness because roughly 80% of the world’s population does not speak English.The introduction identifies English-centric pretraining as the central limitation motivating multilingual models.
- Motivation: Existing responses included releasing dozens of models, each pretrained on a single non-English language.The passage contrasts language-specific models with a more general multilingual solution.
- Contribution: The paper introduces mT5, a multilingual variant of T5 designed to deviate as little as possible from T5’s training recipe.mT5 is presented as a massively multilingual model that continues T5’s approach.
- Contribution: mT5 inherits T5’s general-purpose text-to-text format, empirical-study-based design, and scale.These inherited properties are identified as benefits of retaining the T5 recipe.
2 Background on T5 and C4
T5 uses a unified text-to-text format across generative and classification NLP tasks, including generating literal label text for classification. Its models range from 60 million to 11 billion parameters and were pre-trained on about 1 trillion tokens from the deduplicated, filtered English-language C4 dataset.
- T5: T5 casts all text-based NLP problems in a unified text-to-text format, generating text for both generative tasks and classification labels.For classification, the model outputs the literal label text, such as “positive” or “negative,” rather than a separate class representation.
- T5: 60 million to 11 billion parameters are available across T5’s pre-trained model sizes.These models were pre-trained on around 1 trillion tokens of data.
- C4: About 750GB of English-language text from the public Common Crawl web scrape comprise the C4 pre-training dataset.C4 applies heuristics to extract natural language and performs extensive deduplication.
3 mC4 and mT5
mT5 extends T5 with the multilingual mC4 corpus and training changes for 101 languages. Its design follows T5.1.1 while adapting data sampling and vocabulary to multilingual pre-training.
- mC4 dataset: mC4 extends C4 to multilingual web data by identifying over 100 languages and using all 71 available Common Crawl monthly scrapes.C4 used only the April 2019 scrape, whereas mC4 draws on substantially more source data.
- mC4 dataset: mC4 replaces C4’s English punctuation filter with a line-length requirement, while retaining line deduplication and bad-word page removal.Pages must contain at least three lines with 200 or more characters.
- mC4 dataset: After filtering, mC4 includes languages with at least 10,000 pages, producing text in 107 cld3-defined languages, including six script variants.The six variants represent the same spoken languages in different scripts.
- mT5 model: mT5 closely follows the T5.1.1 recipe, using GeGLU nonlinearities, scaled dmodel and dff in larger models, unlabeled-only pre-training, and no dropout.These architectural and training choices are inherited from the specified T5.1.1 recipe.
- mT5 model: mT5 samples languages with p(L) ∝ |L|^α to boost lower-resource languages while balancing overfitting on scarce languages against underfitting on abundant languages.The paper describes multilingual sampling as a zero-sum tradeoff between low-resource overfitting and high-resource undertraining.
- mT5 model: mT5 increases its vocabulary to 250,000 wordpieces and uses 0.99999 character coverage with byte fallback for uniquely encoding strings across languages.The expanded vocabulary addresses the model’s coverage of over 100 languages and large character sets such as Chinese.
4 Experiments
The experiments evaluate mT5 across six XTREME multilingual tasks using a unified text-to-text formulation and compare performance across model sizes, training settings, and ablations. The results show strong multilingual performance, improved zero-shot behavior with scale, and consistent support for the selected pre-training choices.
- Experimental setup: mT5 is evaluated on six XTREME tasks spanning entailment, reading comprehension, named entity recognition, and paraphrase identification across up to 40 languages.All tasks are cast as text-to-text generation, producing labels, entity tags, or answers directly.
- Experimental setup: The models comprise five sizes from Small (≈300M parameters) through XXL (13B), and are pre-trained for 1 million steps on roughly 1 trillion input tokens.The pre-training setup follows T5, including the same masked-span objective, while mT5 uses a larger vocabulary and about 1/6 as much pre-training as XLM-R.
- Main results: mT5-XXL exceeds state-of-the-art on all classification and QA tasks and reaches 69.2 versus 70.1 on NER.The comparison notes that some competing systems use parallel training data or related-task labeled data, unlike mT5.
- Scaling and training data: For the two largest models, zero-shot and translate-train performance on TyDi QA GoldP is nearly the same, indicating diminishing returns from machine-translated monolingual data as capacity increases.Performance improves with model capacity, while the quality gap between zero-shot, translate-train, and in-language multitask settings narrows.
- Ablations: Every tested ablation lowers average XNLI zero-shot accuracy relative to the mT5-Large baseline, while the line length filter provides a +2 point boost.The findings support the chosen settings and reinforce the value of filtering low-quality Common Crawl pages.
5 Zero-shot generation
mT5’s unconstrained generation causes illegal predictions in zero-shot cross-lingual span selection, including normalization, grammatical adjustment, and accidental translation. Mixing multilingual pre-training into fine-tuning substantially reduces illegal predictions while preserving the general text-to-text framework.
- Illegal predictions: Zero-shot cross-lingual span selection requires generating legal non-English answers even though fine-tuning exposes the model only to English targets.Unlike encoder-based span selection, legality is not a hard constraint for the generative model.
- Illegal predictions: mT5 achieves state-of-the-art zero-shot results on XQuAD, MLQA, and TyDi QA, but still produces illegal spans across model sizes.These errors mainly involve normalization, grammatical adjustment, and accidental translation.
- Accidental translation: Accidental translation occurs across all model sizes and XQuAD languages, especially in mT5-Small and mT5-Base, where at least half of illegal predictions in each language may show it.Partial translation, such as producing one or two English tokens, is more common than translating a full phrase.
- Domain Preserving Training: Restricting inference to legal spans could improve zero-shot metrics but would undermine the general text-to-text interface and fail for open-ended tasks such as summarization.The proposed multilingual-data mixing strategy is designed to apply across zero-shot generation tasks.
- Domain Preserving Training: More than 70% relative reduction in illegal prediction rates is achieved on mT5-Small and mT5-Base by mixing the 101-language unsupervised task into XQuAD fine-tuning at a 1:100 ratio.The method removes sentinel tokens from targets and reduces the language sampling parameter α from 0.3 to 0.1.
6 Conclusion
The paper introduces mT5 and mC4, applies the T5 recipe to multilingual modeling with strong benchmark performance, and addresses illegal zero-shot predictions. It also releases the code and pre-trained datasets to support future multilingual research.
- mT5 and mC4 are introduced as massively multilingual variants of T5 and the C4 dataset.
- The T5 recipe applies straightforwardly to multilingual settings and achieves strong performance across diverse benchmarks.
- The paper characterizes illegal predictions in zero-shot evaluation of multilingual pre-trained generative models and presents a simple avoidance technique.
- The authors release all code and pre-trained datasets used in the paper to facilitate future multilingual research.
- Mixing unlabeled data for only one language at a time is described as contrary to the spirit of multilingual models and zero-shot evaluation.