Source-linked AI summary
Are All Languages Created Equal in Multilingual BERT?
Shijie Wu, Mark Dredze
TL;DR
Prior evaluations left most of mBERT’s 104 languages, especially low-resource languages, insufficiently tested. This paper evaluates within-language representations across NER, POS tagging, and dependency parsing, finding strong resource-dependent disparities and identifying data and pretraining efficiency as priorities.
Problem
Prior evaluations focused mainly on high-resource languages, leaving most of mBERT’s 104 languages untested.
Method
The paper evaluates mBERT on NER for 99 languages and POS tagging and dependency parsing for 54 languages each, using within-language supervised data.
Results
mBERT is comparable to or better than baselines for higher-resource languages but performs much worse for low-resource languages; monolingual BERT performs worse than mBERT there.
Takeaways & Limitations
Better low-resource language models require more data or more data-efficient pretraining techniques.
Takeaways & Limitations
Developing pretrained language models for low-resource languages remains an open challenge.
Abstract
from arXiv · showhide
Multilingual BERT (mBERT) trained on 104 languages has shown surprisingly good cross-lingual performance on several NLP tasks, even without explicit cross-lingual signals. However, these evaluations have focused on cross-lingual transfer with high-resource languages, covering only a third of the languages covered by mBERT. We explore how mBERT performs on a much wider set of languages, focusing on the quality of representation for low-resource languages, measured by within-language performance. We consider three tasks: Named Entity Recognition (99 languages), Part-of-speech Tagging, and Dependency Parsing (54 languages each). mBERT does better than or comparable to baselines on high resource languages but does much worse for low resource languages. Furthermore, monolingual BERT models for these languages do even worse. Paired with similar languages, the performance gap between monolingual BERT and mBERT can be narrowed. We find that better models for low resource languages require more efficient pretraining techniques or more data.
1 Introduction
This paper asks whether mBERT learns equally strong representations across its 104 languages, especially low-resource languages that prior evaluations largely omitted. It evaluates multilingual representation quality across languages and finds substantial resource-dependent differences.
- Prior evaluations emphasized high-resource languages, leaving most of mBERT’s 104 languages—predominantly low-resource languages—untested.
- The paper evaluates mBERT’s representation quality across 99 languages for NER and 54 languages each for POS tagging and dependency parsing.
- mBERT’s bottom 30% languages perform much worse than a non-BERT model on NER.
- Monolingual BERT performs worse than mBERT for low-resource languages, while bilingual BERT improves over monolingual BERT but remains behind mBERT.
- The findings suggest collecting more data or using more data-efficient pretraining techniques to improve low-resource language representations.
2 Related Work
Related work establishes multilingual contextual representations as effective for cross-language learning, while highlighting resource imbalance and unresolved questions about low-resource languages. Prior studies also suggest that data-efficient alternatives and larger multilingual corpora may matter substantially.
- Multilingual BERT-style pretraining has advanced cross-language learning and transfer by training one model on text from multiple languages with a shared vocabulary.
- mBERT’s 104 Wikipedia languages vary considerably in size, despite up-sampling low-resource languages and down-sampling high-resource languages during training.
- Shared subwords correlate with cross-lingual performance, although multilingual representations can also emerge without vocabulary overlap.
- Shared characters and cognates may bias subword learning toward some languages and influence low-resource representation quality.
- Subword models such as fastText and BPEmb are more data-efficient than contextual models when limited text or task supervision is available.
- A multilingual masked language model trained on 2.5TB of CommonCrawl data outperformed a Wikipedia-based model on low-resource Urdu and Swahili for XNLI.
3 Experimental Setup
The experiments measure mBERT representations across many languages and tasks, defining resource level by Wikipedia size and comparing task-specific models with published baselines. Additional pretraining experiments vary language pairing and model configuration.
- 3.1 High/Low Resource Languages: Languages are classified by WikiSize, the log2 size in megabytes of their raw Wikipedia dump; English has 15.5GB and Yoruba 10MB.
- 3.2 Downstream Tasks: The study evaluates NER on 99 languages and POS tagging and dependency parsing on 54 languages using within-language supervised fine-tuning over mBERT representations.
- 3.2 Downstream Tasks: NER uses entity-level F1 on Wikipedia-derived data, while POS uses accuracy and parsing uses LAS and UAS on Universal Dependencies v2.3.
- 3.2 Downstream Tasks: Task models use linear softmax layers for NER and POS, while dependency parsing replaces the graph-based parser’s LSTM with mBERT.
- 3.2 Downstream Tasks: mBERT is compared with published NER, POS, and dependency-parsing systems, though the parsing comparison uses UD v2.2 versus the study’s UD v2.3.
4 Are All Languages Created Equal in mBERT?
mBERT is competitive with baselines for higher-resource languages but degrades sharply for the lowest-resource languages, particularly in NER. The results also show that limited pretraining and supervised data coincide with weaker performance.
- For WikiSize over 6, mBERT is comparable to or better than baselines in all three tasks except NER.
- For NER in the top 10% of languages with WikiSize over 11, mBERT performs worse than the baseline.
- mBERT performance drops significantly below WikiSize 6, representing the bottom 30% of supported languages.
- At the smallest resource level, mBERT shifts from competitive with state-of-the-art to over 10 points behind.
- For NER languages with only 100 labeled sentences, performance drops significantly alongside lower pretraining data.
- The authors caution against expecting mBERT to provide a reliable model across all 104 supported languages.
5 Why Are All Languages Not Created Equal in mBERT?
The analysis links multilingual representation quality to available data and evaluates monolingual, multilingual, and related-language pairing strategies for low-resource languages. mBERT generally benefits low-resource languages relative to monolingual BERT, while data and training efficiency remain limiting factors.
- 5.1 Statistical Analysis: Each analyzed resource factor correlates positively with NER performance, with training size showing the largest association, followed by training vocabulary and WikiSize.The analysis uses single-factor linear models in log scale and reports a statistically significant relationship for each factor.
- 5.2 mBERT vs monolingual BERT: 25% to 50% of subword types in mBERT’s 115K vocabulary appear in the four evaluated low-resource languages.Despite this coverage, the mBERT vocabulary is not optimal: language-specific vocabularies contain more high-frequency items.
- 5.2 mBERT vs monolingual BERT: No monolingual BERT outperforms mBERT except on Latvian POS, while smaller sequence length gives the best monolingual configuration in 5 of 8 cases.The smaller sequence length permits a larger batch size; smaller vocabulary and smaller model settings are mixed or inferior.
- 5.2 mBERT vs monolingual BERT: Higher-resource languages improve with continued pretraining, whereas lower-resource Mongolian performance fluctuates as its cloze-task dev perplexity becomes stuck.For the lower-resource languages, the authors report overfitting to the training set.
- 5.3 mBERT vs Bilingual BERT: Pairing low-resource languages with closely related languages improves downstream performance, with the Afrikaans-Dutch model improving more than the Latvian-Lithuanian model.The authors suggest the larger size of Dutch relative to Afrikaans may explain the difference.
6 Discussion
The discussion finds that the least-resourced languages perform poorly with mBERT and no better with monolingual BERT, while related-language pairing helps but remains behind mBERT. It identifies more data and more efficient pretraining as open needs.
- 6 Discussion: The 30% of languages with the least pretraining resources perform worse than using no pretrained language model at all.The paper cautions against using mBERT alone for these low-resource languages.
- 6 Discussion: Monolingual training does no better for low-resource languages, while pairing closely related languages helps but still lags behind mBERT.Conversely, the highest-resource languages are hurt by massively multilingual joint training and can favor monolingual models.
- 6 Discussion: Developing pretrained language models for low-resource languages remains an open challenge requiring more efficient pretraining, more data, and better use of multilingual corpora.The paper presents these directions as future work.
A Breakdown of experiments in §4
The experiment breakdown comprises NER results and POS-tagging and dependency-parsing results, with the detailed outcomes organized in Tables 5 and 6.
- NER: Table 5 provides the breakdown of the NER experiment in §4.
- POS tagging and parsing: Table 6 provides the breakdown of the POS-tagging and dependency-parsing experiments in §4.