Source-linked AI summary
mGPT: Few-Shot Learners Go Multilingual
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, Tatiana Shavrina
TL;DR
The paper addresses limited multilingual evidence for zero- and few-shot autoregressive language modeling, particularly in low-resource languages. It develops mGPT models spanning linguistically diverse languages and evaluates their in-context and multilingual task performance. The results show broad task performance, including comparability to XGLM1.7B with fewer weights and wider language coverage, while low-resource evaluation remains constrained by limited resources.
Problem
Finetuning requires expensive labeled data and computational resources, motivating zero- and few-shot approaches for cross-lingual and low-resource language scenarios.
Method
The paper develops 1.3B- and 13B-parameter mGPT autoregressive models covering 61 languages from 25 families and evaluates them across multilingual tasks in zero- and few-shot settings.
Results
mGPT1.3B is comparable to XGLM1.7B with fewer weights and broader language coverage, with performance at or above 0.6 for several language families in knowledge probing.
Takeaways & Limitations
The models show potential as a basis for generative pipelines and cross-lingual applications in languages other than English, especially low-resource languages.
Takeaways & Limitations
Limited evaluation resources for low-resource languages constrain analysis of the models’ generalization abilities.
Abstract
from arXiv · showhide
Recent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models. This paper introduces two autoregressive GPT-like models with 1.3 billion and 13 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus. We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism; Deepspeed and Megatron frameworks allow us to parallelize the training and inference steps effectively. The resulting models show performance on par with the recently released XGLM models by Facebook, covering more languages and enhancing NLP possibilities for low resource languages of CIS countries and Russian small nations. We detail the motivation for the choices of the architecture design, thoroughly describe the data preparation pipeline, and train five small versions of the model to choose the most optimal multilingual tokenization strategy. We measure the model perplexity in all covered languages and evaluate it on the wide spectre of multilingual tasks, including classification, generative, sequence labeling and knowledge probing. The models were evaluated with the zero-shot and few-shot methods. Furthermore, we compared the classification tasks with the state-of-the-art multilingual model XGLM. source code and the mGPT XL model are publicly released.
1 Introduction
The paper motivates multilingual zero- and few-shot autoregressive language modeling as an alternative to data- and resource-intensive finetuning. It introduces mGPT to expand multilingual coverage, especially for CIS and under-resourced Russian languages, and reports broad evaluation results.
- Finetuning requires extensive labeled data and additional computational resources, while collected labels can be expensive and models may generalize inconsistently.
- Zero-shot and few-shot learning perform tasks without parameter updates, supporting cross-lingual knowledge transfer and low-resource language scenarios.
- mGPT provides 1.3B- and 13B-parameter multilingual autoregressive models covering 61 languages from 25 language families.
- The paper targets CIS languages and under-resourced languages of small peoples in Russia while analyzing intrinsic and extrinsic tasks against contemporaneous generative models.
- mGPT1.3B is comparable to XGLM1.7B with fewer weights and broader language coverage, while additional demonstrations may degrade performance and hate-speech detection remains difficult.
2 Related Work
Related work traces a shift from multilingual Transformer models toward large generative and prompt-based language models. Prior multilingual GPT-style research established cross-lingual evaluation, including XGLM's coverage of 30 languages.
- Massively multilingual models including mBERT, XLM-R, RemBERT, mBART, and mT5 have advanced multilingual NLP across multiple languages.
- Large-scale generative language models are shifting NLP from pretrain-and-finetune toward prompt-based learning with demonstration examples.
- XGLM extended GPT-style multilingual research to 30 languages and evaluated in-context abilities in multilingual scenarios.
3 Method
mGPT uses a typologically weighted multilingual corpus, a staged data-cleaning pipeline, and GPT-3-inspired architecture with sparse attention. Tokenization experiments select DEFAULT, which is then used for the 1.3B and 13B models.
- Pretraining Data: The pretraining corpus covers languages selected from cross-lingual benchmarks, including underrepresented Russian and CIS languages.The selection aims to narrow the gap between high-resource and low-resource languages.
- Pretraining Data: The data pipeline collects Wikipedia and C4 documents, deduplicates texts by 64-bit hashes, and filters documents by compression rate and quality.Acceptable compression rates are empirically set from ×1.2 to ×8 before binary quality classification.
- Tokenization: Five tokenization strategies vary the treatment of case, numbers, arithmetic, punctuation, whitespace, and character-level units.The strategies are evaluated using perplexity on a held-out Wikipedia sample.
- Model Pretraining: mGPT replicates GPT-3 parameter configurations and alternates dense and sparse attention layers, using GPT-2 sources and Megatron-LM and DeepSpeed frameworks.The models are trained for 600k steps on 400B tokens using large GPU clusters.
- Tokenization: The DEFAULT tokenization strategy achieves the best results, outperforming the other strategies by up to 2.5 perplexity points.The authors select DEFAULT for pretraining mGPT1.3B and mGPT13B.
- Evaluation: Language-wise and family-wise perplexities are reported, with lower scores indicating better language-modeling performance.The figures organize results by individual languages and by family-averaged scores.
4 Experiments
The experiments evaluate mGPT with perplexity, zero-shot and few-shot prompting, sequence labeling, knowledge probing, and generation across multilingual settings. Results show stronger language modeling for the larger model, mixed downstream-task performance, and important sensitivity to language and prompting conditions.
- Language Modeling: Perplexity was measured on held-out language sets, with statistical analyses of script, corpus size, and model size.The analyses used Mann-Whitney U tests for script and model size, and Pearson correlation for corpus size.
- Language Modeling: The mGPT13B model achieved perplexities between 2 and 10 for most languages, while mGPT1.3B was consistently higher.Higher perplexities up to 20 occurred for only seven languages.
- Language Modeling: Non-Latin languages received lower average scores, while mGPT13B performed better than mGPT1.3B in this setting.The authors also report a positive correlation between corpus size and perplexity for particular languages, potentially reflecting low domain diversity.
- Classification: Classification results were sensitive to prompt design, and adding demonstrations could degrade performance for both mGPT and XGLM.Reproduction differences for PAWSX and XNLI were attributed to potential differences in translated prompts.
- Sequence Labeling: In sequence labeling, mGPT1.3B outperformed mGPT13B on all NER languages and nearly all POS languages, but both remained below fine-tuned models.For NER, 4-shot performance significantly exceeded random guessing; POS accuracy averaged 0.24 for mGPT1.3B and 0.21 for mGPT13B.
- Knowledge Probing: Knowledge probing scores were at least 0.6 for several language families, while Uralic, Slavic, Koreanic, and Afro-Asiatic languages scored below 0.5.Scaling usually improved high-resource-language performance by up to 5 points, with no significant improvement in other languages.
5 Discussion
The discussion finds that multilingual autoregressive LM performance depends on model size, language resources, and prompting choices. It also identifies unresolved data, tokenization, and sequence-labeling limitations while describing continued pretraining efforts.
- Empirical Results: Scaling improves language modeling for all languages but does not improve downstream or probing tasks, and smaller models may better encode linguistic information.
- Lack of Data: Low-resource evaluation is constrained by limited high-quality data and evaluation resources, while performance correlates with the amount of language-specific pretraining data.
- Empirical Results: mGPT1.3B classification results are on par or better than XGLM1.7B while covering twice as many languages, but it underperforms on several multi-class and probing tasks.
- Tokenization: The optimal multilingual tokenization method and vocabulary size remain open questions because experiments used a fixed vocabulary size.
- Empirical Results: Increasing demonstrations can decrease performance on some downstream tasks, while zero-shot and few-shot prompting may remain near random guessing on particular tasks.
- Empirical Results: Prompting is unstable across languages, and models may assign higher probabilities to the most frequent input tag in sequence labeling.
6 Conclusion
The conclusion presents mGPT1.3B and mGPT13B as multilingual autoregressive models spanning 61 languages and 25 families, with particular attention to underrepresented CIS and Russian languages. It reports potential for low-resource generative applications while acknowledging room for further improvement.
- Conclusion: mGPT1.3B and mGPT13B cover 61 languages from 25 linguistically diverse language families.
- Conclusion: The models target economically endangered and underrepresented CIS and low-resource languages through zero-shot and few-shot evaluation across language modeling, NLU, knowledge probing, and social-bias tasks.
- Conclusion: The authors report potential for developing generative pipelines and supporting cross-lingual knowledge transfer and annotation projection for underrepresented languages.
7 Ethical Statement and Social Impacts
The ethical discussion addresses the computational cost and social risks of multilingual language models. It reports measured emissions, notes potential benefits from multilingual pretraining, and acknowledges risks from bias and misuse.
- Social Impacts: The project addresses low-resource languages but remains relevant to the unresolved resource gap across languages.
- Energy Efficiency and Usage: Sparse attention was used to reduce computational resources for large-scale LM pretraining, which is energy-intensive and expensive.
- Energy Efficiency and Usage: 15.9k kg and 47.5k kg of CO2 were emitted by pretraining mGPT1.3B and mGPT13B, respectively.
- Energy Efficiency and Usage: Multilingual pretraining can reduce data annotation and energy consumption, while pruning and distillation can reduce inference costs.
- Social Risks: Pretraining stereotypes can produce representation biases and harmful stereotypes involving gender, race, religion, or sexual orientation.
- Social Risks: The models can be misused to generate propaganda and targeted manipulation across all supported languages, although adversarial defenses and detection models may mitigate risks.