Source-linked AI summary
SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, Annie En-Shiun Lee
TL;DR
Multilingual NLU benchmarks often omit low-resource languages, motivating SIB-200, an open topic-classification dataset spanning over 200 languages and dialects. The paper builds it from Flores-200 annotations and evaluates supervised, transfer, and prompting settings, finding persistent high-resource versus low-resource gaps and stronger transfer than prompting for many languages.
Problem
Existing NLU benchmarks cover relatively few languages, limiting evaluation evidence for truly low-resource languages.
Method
The paper extends English sentence-level topic annotations from Flores-200 to the corpus’s other languages and evaluates multilingual models through supervised, transfer, and prompting settings.
Results
The evaluations show a large performance gap between high- and low-resource languages, while transfer from high-resource languages performs much better than prompting for many languages.
Takeaways & Limitations
SIB-200 enables more inclusive multilingual evaluation across languages often missing from NLU benchmarks.
Takeaways & Limitations
The benchmark has 1,004 examples, and non-English labels may suffer translationese effects that slightly reduce performance.
Abstract
from arXiv · showhide
Despite the progress we have recorded in the last few years in multilingual natural language processing, evaluation is typically limited to a small set of languages with available datasets which excludes a large number of low-resource languages. In this paper, we created SIB-200 -- a large-scale open-sourced benchmark dataset for topic classification in 200 languages and dialects to address the lack of evaluation dataset for Natural Language Understanding (NLU). For many of the languages covered in SIB-200, this is the first publicly available evaluation dataset for NLU. The dataset is based on Flores-200 machine translation corpus. We annotated the English portion of the dataset and extended the sentence-level annotation to the remaining 203 languages covered in the corpus. Despite the simplicity of this task, our evaluation in full-supervised setting, cross-lingual transfer setting and prompting of large language model setting show that there is still a large gap between the performance of high-resource and low-resource languages when multilingual evaluation is scaled to numerous world languages. We found that languages unseen during the pre-training of multilingual language models, under-represented language families (like Nilotic and Altantic-Congo), and languages from the regions of Africa, Americas, Oceania and South East Asia, often have the lowest performance on our topic classification dataset. We hope our dataset will encourage a more inclusive evaluation of multilingual language models on a more diverse set of languages. https://github.com/dadelani/sib-200
1 Introduction
Multilingual evaluation remains concentrated in relatively few languages, limiting evidence for low-resource languages. SIB-200 addresses this gap with a large-scale topic-classification benchmark and evaluations spanning transfer and language-model prompting.
- Benchmark datasets often cover only a few tens of languages, limiting large-scale evaluation, especially for truly low-resource languages.
- SIB-200 is an open-sourced topic-classification benchmark based on Flores-200, extending English sentence annotations across the corpus’s other languages.
- The evaluation compares multilingual-model transfer with large-language-model prompting across numerous languages.
- Cross-lingual transfer from high-resource languages performs much better than prompting large language models for many languages.
2 SIB-200 dataset
SIB-200 converts Flores-200 into a sentence-level topic-classification benchmark through English annotation, multilingual extension, quality control, and category filtering. The released benchmark contains 1,004 sentences after exclusions and label consolidation.
- Data source: SIB-200 is a topic-classification dataset covering over 200 languages and dialects, built from the multi-way parallel Flores-200 corpus.Flores-200 contains 3,001 sentences across DEV, DEVTEST, and TEST sets, although TEST was not released.
- Annotation: Four native English annotators labeled 2,009 DEV and DEVTEST sentences using 15 categories, including ten original categories and five added categories.The annotation took about two weeks and approximately 33 hours in total.
- Quality control: Annotation agreement was moderate, with a Fleiss Kappa score of 0.44 among the four annotators.
- Quality control: Final labels used majority voting, with lead-annotator adjudication for 314 conflicting sentences.A single label was initially assigned to 1,695 sentences before adjudication restored labels for all 2,009 annotated sentences.
3 Experimental setup
The experiments organize languages by region, family, resource level, and pre-training coverage, then compare lightweight models, multilingual PLMs, adapted models, and prompted LLMs. Evaluation includes fully supervised, cross-lingual transfer, and zero-shot prompting settings.
- Language categorization: Languages are grouped by geography, language family, multilingual-PLM coverage, and Joshi’s web-resource classification to structure result analysis.
- Language categorization: SIB-200 covers 21 language families, including 79 Indo-European, 35 Atlantic-Congo, 21 Afro-Asiatic, 21 Austronesian, and 11 Turkic languages.
- Text classification models: The model suite includes an MLP, fine-tuned multilingual and regional PLMs, MAFT-adapted models, and prompted GPT-3.5-Turbo and GPT-4.The MLP uses word n-grams or XLM-R tokens, while multilingual PLMs include XLM-R and Glot-500.
- Experimental settings: Fully supervised experiments train and evaluate within each language across 205 languages, assuming labeled target-language data.
- Experimental settings: Cross-lingual transfer fine-tunes XLM-R on English, French, Arabic, or Simplified Chinese and evaluates other languages, assuming labeled data in a few high-resource languages.
- Experimental settings: Zero-shot prompting evaluates GPT-3.5 and GPT-4 on 205 languages using an English classification template without target-language labeled data.
4 Results
Across supervised, transfer, prompting, and model analyses, performance varies substantially by language resources, family, region, pretraining coverage, and script. XLM-R generally leads, but low-resource and underrepresented languages remain difficult, while targeted regional or feature-based approaches sometimes help.
- Baseline results: 92.1% accuracy on English falls to 75.9% on average for other languages under fully supervised XLM-R fine-tuning.Cross-lingual transfer and zero-shot prompting reduce performance further.
- Language families: Atlantic-Congo, Nilotic, Mande, Aymaran, and Quechuan languages remain below 65% accuracy even with the best fully supervised model.These families also show larger fully supervised-to-transfer gaps, indicating poorer semantic alignment.
- Regions and Joshi’s classes: Languages in Joshi’s classes ≥3 reach approximately 90% accuracy, while African languages perform worst within the same class.On class 0, African languages are at least 20% below languages from other continents.
- Model comparisons: XLM-R performs best for 16 of 22 language families, whereas simple MLP n-gram features outperform transformer models on six families.Glot-500 surpasses XLM-R only for Sino-Tibetan languages, where it still does not beat the MLP baseline.
- Pretraining corpus size: A 0.1GB XLM-R pretraining corpus already yields over 80% accuracy for almost all languages, with performance generally increasing before saturating above 1GB.The corpus-size relationship is evaluated in the fully supervised setting.
- Script effects: N-gram features are more robust across scripts than word features from the XLM-R tokenizer.MLP-XLM-R performance generally tracks XLM-R, suggesting script preferences persist independently of model complexity.
- Fine-tuning and prompting: Zero-shot prompting consistently underperforms fine-tuned methods, although GPT-4 outperforms GPT-3.5-turbo in 157 of 205 languages.Prompting performs well for only a few families, including Indo-European, Uralic, Japonic, and Koreanic.
- Region-specific pretraining: Region-specific models outperform XLM-R for some linguistically or geographically related languages, while synthetic-data MAFT improves African-language accuracy by up to 7.9 points.AfroXLMR-76 gains +7.9 for Nilotic, +4.5 for Mande, and +7.4 for Atlantic-Congo; uncovered scripts such as N’ko and Tamazight do not improve.
5 Related Work
Prior multilingual evaluation datasets span multiple tasks, but SIB-200 addresses a text-classification gap by covering 200 languages and broader topics and domains.
- 5 Related Work: Multilingual evaluation datasets cover tasks including part-of-speech tagging, named entity recognition, entity linking, natural language inference, text classification, and machine translation.These efforts have contributed to cross-lingual and multilingual NLP.
- 5 Related Work: Taxi1500 covers 1500 languages but focuses on the religious domain because its data comes from the Bible.It is described as the largest and most recent multilingual text-classification dataset among the listed works.
- 5 Related Work: SIB-200 addresses a multilingual text-classification gap by covering 200 languages and a broader range of topics and domains.The dataset builds on prior multilingual text-classification efforts while broadening domain coverage beyond religious data.
6 Conclusion
The paper creates SIB-200, an open-sourced topic-classification benchmark for 200 languages and evaluates it across supervised, transfer, and prompting settings. It finds performance differences across language groups and analyzes which categories perform poorly.
- 6 Conclusion: SIB-200 is an open-sourced benchmark dataset for topic classification in 200 languages and dialects.It targets the lack of NLU evaluation datasets, especially for low-resource languages.
- 6 Conclusion: The evaluation covers fully supervised, cross-lingual transfer, and large-language-model prompting settings.The study also groups languages by families, regions, resource class, and multilingual-model coverage.
- 6 Conclusion: The benchmark provides performance insights across language families, geographical regions, web-resource classes, and multilingual pre-trained-language-model coverage.These groupings are used to identify categories with poor performance on the benchmark.
7 Limitations
The paper’s limitations include benchmark size, translationese effects, and evaluation of only three multilingual models; it also describes the language grouping and African adaptation setup used in the analysis.
- Data size: The benchmark contains 1,004 instances, and the authors identify its size as a limitation.They nevertheless consider it important for languages lacking datasets suitable for topic-classification annotation.
- Translationese effect: Non-English labeled data are based on human translation and may suffer from translationese effects, including a slight performance drop.This constrains how directly results transfer to naturally occurring non-English text.
- Few PLMs evaluated: The evaluation is limited to three multilingual models: XLM-R-base, XLM-R, and Glot-500.The authors note that XLM-R may not be the best multilingual encoder, while newer models may cover similar languages.
- Language categorization: Languages are grouped by geographical region, language family, multilingual-PLM coverage, and Joshi’s resource classification.The dataset includes 21 language families, while 128 languages are classified as low-resource, 30 as mid-resource, and 39 as high-resource.
- Region-specific pre-training: The African adaptation extends XLM-R to 61 languages with at least 10MB of monolingual data and adds machine-translated data for 34 languages with less data.The resulting AfroXLMR-76 model is pretrained on 21GB of data.
C SIB-200 English dataset performance using 7 or 14 labels
On the English SIB-200 dataset, fine-tuned XLM-R performs substantially better with seven labels than with fourteen labels.
- English dataset performance: 92.5% accuracy is achieved with 7 labels, compared with 82.3% accuracy with 14 labels.Both results come from fine-tuning XLM-R on SIB-200.
D Overall result
Overall results are reported across all languages in the benchmark.
- Table 8 reports performance across all languages for the evaluated text classification models and settings.
E Results by categorization of regions
Regional performance is summarized through baseline box plots, enabling comparison across geographic regions.
- Figure 6 presents baseline results for each represented region as box plots.
F African languages result
The African-language analysis evaluates script-focused vocabulary augmentation and broader model performance across African languages. Script augmentation improves N’ko and Tfng results substantially, but performance remains below MLP results.
- Vocabulary augmentation adds script-specific tokens and applies MAFT to produce AfroXLMR-76-script.The procedure trains a tokenizer for N’ko, Tifinagh, and Tamazight, adds the top 20K tokens, and performs MAFT on XLM-R.
- +18 points improvement is reported for languages using Nkoo and Tfng after vocabulary augmentation.Despite this gain, performance remains lower than using MLP for these languages.
- The African-language results are summarized in Table 9, while script-based results are presented in Table 7.
- Table 8 compares fully supervised, cross-lingual transfer, and zero-shot prompting settings across languages, including four source languages.The source languages are English, French, Chinese, and Arabic.