Source-linked AI summary
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages
Ayyoob Imani, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André F. T. Martins, François Yvon, Hinrich Schütze
TL;DR
Existing multilingual LLMs cover only a small fraction of the world’s languages, especially neglecting low-resource languages. The paper constructs Glot500-c and uses continued pretraining to create Glot500-m for 511 languages, finding large improvements over XLM-R for both head and tail languages. It further reports that representation quality reflects multiple factors rather than a single determinant.
Problem
Existing multilingual LLMs support only about 100 of the world’s 7000 languages, while data scarcity leaves tail languages frequently ignored.
Method
The authors collect and clean Glot500-c, then use it for continued pretraining of XLM-R to create Glot500-m covering 511 languages.
Results
Glot500-m shows large improvements over XLM-R for both head and tail languages across diverse tasks, with representation quality shaped by corpus size, script, related-language help, and model capacity.
Takeaways & Limitations
Multilingual LLM development can expand beyond a small set of high-resource languages toward broader support for the world’s languages and cultures.
Takeaways & Limitations
The study performs no comprehensive hyperparameter search, uses a comparatively small model relative to current very large models, and retains some data noise.
Abstract
from arXiv · showhide
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, "help" from related languages and the total capacity of the model. Our work addresses an important goal of NLP research: we should not limit NLP to a small fraction of the world's languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures. Code, data and models are available at https://github.com/cisnlp/Glot500.
1 Introduction
The paper shifts multilingual LLM scaling from deepening performance in roughly 100 high-resource languages toward supporting hundreds of predominantly low-resource languages. It builds and evaluates Glot500-m using the Glot500-c corpus, reporting stronger tail-language performance and multiple factors underlying representation quality.
- Existing multilingual LLMs support only about 100 of the world’s 7000 languages, leaving tail languages underserved by language technologies.
- Glot500-m horizontally scales multilingual coverage to 511 languages, most of them low-resource, to address this gap.
- Glot500-c contains 511 languages and 534 language-scripts selected after cleaning and deduplication, with more than 30,000 sentences per included language-script.
- The authors train Glot500-m through vocabulary extension and continued pretraining of XLM-R with the masked-language-modeling objective.
- Glot500-m outperforms XLM-R-B on tail languages by a large margin while performing comparably or better on head languages across diverse evaluation tasks.
- The analysis attributes multilingual representation quality to a combination of corpus size, script, related-language help, and total model capacity rather than one factor.
2 Related Work
Prior work improves multilingual modeling and extends language coverage, but often remains limited by language count, data availability, computational demands, or restricted evaluation. The paper positions horizontally scaled models as a route toward broader tail-language support and cross-lingual transfer.
- Multilingual pretraining methods such as MLM, discriminative pretraining, and parallel data primarily benefit languages with large corpora.
- Prior adaptation work includes vocabulary extension, MLM or TLM objectives, adapters, bilingual lexicons, and parameter-efficient fine-tuning for limited sets of languages.
- Some models are trained from scratch for small groups of African or Indic languages, while a concurrent 517-language African model used a 42 GB corpus but was not made available.
- Large-scale web-crawl approaches create resources for many tail languages but require substantial computation and native-speaker involvement, limiting accessibility outside large corporations.
- Horizontally scaled multilingual LLMs may transfer knowledge from high-resource to low-resource languages, although this paper does not make transfer its main focus.
3 Glot2000-c
Glot2000-c aggregates and cleans multilingual data from diverse sources, then selects Glot500-c for model training using language-script identification, noise filtering, deduplication, and a 30,000-sentence threshold.
- 3.1 Data Collection: The authors collect more than 700GB of text in 2266 languages from about 150 sources spanning religious, news, and scientific domains.
- 3.1 Data Collection: They detect scripts sentence by sentence and treat each language-script as a separate entity because some languages use multiple scripts.
- 3.3 Ngram LMs and Language Divergence: A 3-gram character-level language model provides perplexity-based divergence used to filter noisy data and study effects of related languages.
- 3.4 Data Cleaning: Cleaning combines corpus-level checks for language or script mismatches with chunk-level filters for repetition, special characters, insufficient length, and duplicate text.
- 3.5 Training Data: Glot500-c: The authors retain language-scripts with at least 30,000 sentences’ as a relatively high safe threshold for effective pretraining and broad low-resource coverage.
- 3.5 Training Data: Glot500-c: The corpus is divided into train, development, and test data, with parallel Bible verses added when available to facilitate cross-lingual evaluation.
4 Glot500-m
Glot500-m extends XLM-R’s vocabulary and continues masked-language-model pretraining on language-script-balanced data. It uses the same transformer size as XLM-R-B but a larger vocabulary, while tokenization itself has little observed effect on downstream performance.
- 4.1 Vocabulary Extension: The authors train a 250K-token SentencePiece vocabulary on Glot500-c and merge its tokens with XLM-R’s vocabulary, favoring tail languages during sampling.
- 4.1 Vocabulary Extension: The percentage of changed tokens ranges from 0.2% to 50% for head languages, but it has no observed relationship with downstream performance changes.
- 4.2 Continued Pretraining: Glot500-m is created by continued XLM-R-B pretraining with the masked-language-modeling objective using batches sampled across language-scripts.
- 4.2 Continued Pretraining: The model is trained for two weeks on eight NVIDIA RTX A6000 GPUs, with checkpoints selected by downstream-task performance.
5 Experimental Setup
Glot500-m is evaluated with complementary intrinsic and downstream multilingual tests spanning representation quality, sequence labeling, retrieval, and classification. The setup combines human-labeled and label-free evaluations to cover languages lacking gold data.
- Evaluation strategy: The overall validation strategy mixes human labels with methods applicable to many languages without gold data.This design addresses the scarcity of manually labeled evaluation data for tail languages.
- Evaluation tasks: The evaluation suite covers pseudoperplexity, roundtrip alignment, sequence labeling, sentence retrieval, and text classification across many language-scripts.Roundtrip alignment provides a multilingual representation measure without requiring human gold data.
- Evaluation tasks: Roundtrip alignment follows word alignments across several language-scripts and measures the percentage of paths that return to the starting word.Three intermediate language-scripts are sampled, and results are averaged over five intermediate-point selections.
- Evaluation tasks: Sequence labeling evaluates zero-shot transfer for NER and POS by fine-tuning on English and testing on WikiANN and Universal Dependencies languages.The setup uses English training because labeled training data is unavailable for some target languages.
- Evaluation tasks: Sentence retrieval uses English-aligned Tatoeba and Bible sentences, cosine similarity between layer-8 average word embeddings, and top10 accuracy.The evaluation uses up to 1000 Tatoeba sentences and 500 Bible sentences.
- Evaluation tasks: Text classification uses Taxi1500 gold data across 354 supported language-scripts, with English fine-tuning followed by zero-shot testing.The task contains six classes and applies early stopping on English development data.
6 Experiments
Across aggregate evaluations, Glot500-m generally improves over XLM-R for tail language-scripts and often for head language-scripts, while gains vary with corpus size, script coverage, related languages, and model capacity. Training improves rapidly early and peaks at 480K steps, but the analysis remains non-conclusive for some language differences.
- 6.1 Results: Glot500-m outperforms XLM-R-B across head and tail language-scripts on nearly all tasks, except POS for head languages, with especially large tail-language margins.It also outperforms XLM-R-L on all tail-language tasks and on three head-language tasks.
- Training progression: Training performance improves rapidly at first, slows later, and reaches its average downstream-task peak at 480K steps.The slowdown is particularly pronounced for tail languages in sentence retrieval.
- 6.4 Analysis across Language-Scripts: The gain over XLM-R-B is large for some tail language-scripts but small or negative for others, reflecting variation across four evaluated tasks.Table 6 explicitly contrasts the five highest-gain and five lowest-gain tail language-scripts.
- 6.4 Analysis across Language-Scripts: Uncovered scripts receive large improvements because XLM-R maps them to unknown tokens, while corpus size also contributes to gains for tail languages.Dhivehi in Thaana script is given as an example of a script absent from XLM-R pretraining.
- 6.5 Languages with Multiple Scripts: For languages with multiple scripts, the script with more collected data can perform better; kaa_Cyrl outperforms kaa_Latn by 30% with about three times as much data.The comparison concerns sentence retrieval performance for Kara-Kalpak.
- Analysis: The analysis concludes that model quality depends on interacting factors rather than a single determinant, including corpus size, script, related-language support, and capacity.The authors note that the appropriate capacity may differ across language pairs in a horizontally scaled model.
- 6.7 Corpus Size: Corpus size correlates moderately with SentRetr Bible performance, with Pearson’s r=.34 for a language and r=.44 when including its nearest neighbors.The latter correlation is reported for both k=3 and k=4 nearest neighbors.
7 Conclusion and Future Work
The paper constructs and evaluates Glot500-c and Glot500-m to extend language-model coverage to hundreds of usually neglected languages. It reports improvements for both head and tail languages and identifies multiple factors shaping multilingual representations.
- Glot500-c is a large, cleaned corpus of hundreds of usually neglected tail languages used to create Glot500-m.
- Glot500-m is evaluated on six tasks that cover almost all languages.
- Glot500-m shows large improvements over XLM-R for both head and tail languages.
- Representation quality reflects a combination of corpus size, script, related-language support, and total model capacity.
- The authors plan to study larger models, resource-efficient deployment, alternative pretraining methods, and more tail-language tasks.
Limitations
The authors identify limited hyperparameter exploration, comparatively small model size, and residual data noise as limitations.
- No comprehensive hyperparameter search was performed because training multiple models was costly.
- Glot500-m is comparatively small relative to current very large models, and some data noise remains despite cleaning efforts.
Ethics Statement
The project identifies limitations involving data bias review and data reuse rights. It also documents methodological details for evaluating language divergence and typological relatedness.
- Ethics Statement: The data could not be thoroughly examined for all languages, so the team cannot confirm the absence of discrimination based on race or sexuality.The corpus was used as textual data, and potential bias in generated outputs is left for future research.
- Ethics Statement: Some sources permit research use without permitting redistribution, so data from those sources is omitted from the published Glot2000-c version.The project notes that source copyright statements vary.
- Language Divergence: Language divergence is computed using the maximum perplexity in both language directions to obtain a symmetrical measure.The method accounts for differences in language complexity that can make one direction substantially lower than the other.
- Runs and Data: The evaluation uses the same training and testing data as Glot500-m, limits training to 100,000 sentences per language-script, and evaluates seven n-gram models.The models use KenLM with interpolated modified Kneser-Ney smoothing and n values from 3 to 9.
- Results: Classification accuracy decreases as typological family level and the number of nearest neighbors increase.The reported results use a 3-gram model and exclude languages without a family or another family member in the data.
C List of data sources
The project combines resources from numerous repositories and datasets to support its multilingual corpus and model. The listed materials span language-specific, multilingual, translation, and web-based sources.
- Data sources: The listed repositories include AI4Bharat and the AIFORTHAI-LotusCorpus, with links supplied for those resources.
- Data sources: The data-source list includes resources such as AI4Bharat, AfriBERTa, AfroMAFT, Anuvaad, AraBench, Bloom, CC100, and CCNet.
- Data sources: Additional sources include multilingual and language-specific corpora such as Clarin, DART, Earthlings, Flores200, GiossaMedia, Glosses, Habibi, and IndicNLP.
- Data sources: The collection also draws on Lyrics, MC4, MTData, MaCoCu, Masakhane community resources, NLLB_seed, OPUS, OSCAR, ParaCrawl, and parallel corpora for Ethiopian languages.
- Language inventory: The appendices organize the language inventory used to train Glot500-m across three tables.
D Results for Each Task and Language
The paper reports detailed results for every evaluated task and language in separate appendix tables. These tables cover retrieval, tagging, classification, and alignment tasks.
- Results by task: Sentence Retrieval Tatoeba results are reported in Table 14.
- Results by task: Sentence Retrieval Bible results are reported in Tables 15 and 16.
- Results by task: NER and POS results are reported in Tables 17 and 18, respectively.
- Results by task: Text Classification results are reported in Tables 19 and 20.
- Results by task: Round Trip Alignment results are reported in Tables 21 and 22.
E Perplexity Results for all Languages
The appendices provide language-level evaluation results for Glot500-m and its comparison models. They cover task metrics and perplexity across the languages included in the study.
- Perplexity results: Tables 23–25 present perplexity for all languages covered by Glot500-m.
- Task results: Tables 14–22 report XLM-R-B, XLM-R-L, and Glot500-m results across sentence retrieval, NER, POS, text classification, and round-trip alignment.
- Metrics: The appendix table captions identify Top10 accuracy for sentence retrieval and F1 for NER, POS, and text classification.
- Metrics: Round Trip Alignment is evaluated using accuracy in Tables 21 and 22.