Source-linked AI summary
How multilingual is Multilingual BERT?
Telmo Pires, Eva Schlinger, Dan Garrette
TL;DR
The paper asks whether M-BERT can generalize task knowledge across languages despite being trained on separate monolingual corpora. Using zero-shot transfer and probing experiments, it finds robust multilingual representations that support cross-script transfer, while revealing weaknesses for typologically divergent and transliterated targets.
Problem
The paper investigates how far M-BERT representations trained on monolingual corpora generalize across languages.
Method
The authors fine-tune M-BERT on supervised task data in one language and evaluate it in another, using probing experiments to examine transfer mechanisms.
Results
M-BERT generalizes across scripts and low lexical overlap, with transfer strongest between typologically similar languages and hidden representations showing a shared language-agnostic subspace.
Takeaways & Limitations
M-BERT creates multilingual representations without explicit multilingual training, but those representations have systematic deficiencies for typologically divergent and transliterated targets.
Takeaways & Limitations
M-BERT is not able to effectively transfer to transliterated targets and is outperformed by prior work in monolingual-only and code-switched supervision scenarios.
Abstract
from arXiv · showhide
In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.
1 Introduction
The paper investigates whether M-BERT’s representations generalize across languages through zero-shot transfer. Its probing results show strong cross-lingual generalization, including across scripts, while revealing dependence on typological similarity.
- Research question: The study examines how M-BERT representations generalize across languages using a model pretrained on monolingual Wikipedia corpora from 104 languages.M-BERT is evaluated through cross-lingual transfer rather than language-specific retraining.
- Method: Zero-shot transfer fine-tunes M-BERT on task annotations in one language and evaluates the task in another language.This setup exposes how information generalizes across languages.
- Findings: M-BERT transfers across different scripts with effectively zero lexical overlap, showing that its multilingual representations extend beyond vocabulary matching.The results indicate that transfer can map learned structures onto new vocabularies.
- Findings: Transfer works best between typologically similar languages, while different word orders remain a systematic difficulty.The model maps structures onto new vocabularies but does not appear to learn systematic transformations for different word orders.
2 Models and Data
M-BERT is a 12-layer Transformer trained on 104 languages with a shared word-piece vocabulary, and the experiments evaluate zero-shot NER and POS transfer across multilingual datasets.
- Models: M-BERT is a 12-layer Transformer trained on Wikipedia pages from 104 languages with a shared word-piece vocabulary.Unlike EN-BERT, it is not trained only on English data.
- Models: M-BERT has no input-language marker or explicit mechanism encouraging translation-equivalent representations to align.Its multilingual behavior therefore arises without those explicit design mechanisms.
- Evaluation: For NER and POS, the model is fine-tuned with a sequence-tagging architecture that feeds BERT activations into a final prediction layer.Training minimizes cross-entropy loss, and split words use the first word-piece prediction.
- Data: NER experiments use CoNLL data covering Dutch, Spanish, English, and German, plus an in-house dataset spanning 16 languages.The CoNLL results report zero-shot performance across language pairs.
- Data: POS experiments use Universal Dependencies data for 41 languages, with reported zero-shot results for four European languages.The evaluation sets come from Zeman et al. (2017).
3 Vocabulary Memorization
The experiments test whether M-BERT’s transfer depends on shared vocabulary pieces. M-BERT remains effective across low-overlap and different-script pairs, but performance varies with typological similarity and target-language conditions.
- Vocabulary overlap: The study measures entity word-piece overlap to test whether zero-shot NER transfer is mainly vocabulary memorization.Overlap is the fraction of shared entity word pieces between training and evaluation languages.
- Vocabulary overlap: M-BERT scores 40%–70% F1 for language pairs with almost no lexical overlap, whereas EN-BERT scores are near zero across different scripts.M-BERT performance is flat across a wide overlap range, unlike EN-BERT’s dependence on overlap.
- Cross-script transfer: An Urdu-fine-tuned M-BERT model achieves 91% POS accuracy on Hindi despite never seeing POS-labeled Devanagari words.Urdu uses Arabic script, while Hindi uses Devanagari script.
- Cross-script transfer: Cross-script transfer is less accurate for some pairs, such as English–Japanese, indicating that M-BERT does not generalize equally well across languages.The paper links this variation to differences such as word order.
4 Encoding Linguistic Structure
M-BERT’s multilingual representation maps linguistic structures across languages, with transfer improving as languages become more typologically similar. However, it transfers less effectively when target languages differ in word order or use transliteration.
- 4.1 Effect of language similarity: M-BERT’s zero-shot POS accuracy improves as languages share more WALS features, although it remains effective for low-similarity pairs relative to EN-BERT.The experiments compare POS accuracy with the number of shared grammatical-ordering features.
- 4.2 Generalizing across typological features: Zero-shot POS transfer performs best between languages sharing word-order features such as subject–object–verb and adjective–noun order.The results indicate that M-BERT maps learned structures onto new vocabularies but does not systematically transform them for different word orders.
- 4.3 Code switching and transliteration: M-BERT can generalize from monolingual supervision to code-switched Hindi/English inputs when Hindi is written in Devanagari.Performance with monolingual Hindi and English training is comparable to training on code-switched data for script-corrected inputs.
- 4.3 Code switching and transliteration: M-BERT does not effectively transfer to transliterated target text, indicating a limitation in cross-script generalization despite its multilingual representations.The authors suggest that language-model pre-training on the particular target language supports transfer, while prior systems used explicit transliteration signals.
5 Multilingual characterization of the feature space
The study tests whether sentence-level transformations in M-BERT’s feature space are consistent across translation pairs. Nearest-neighbor retrieval shows shared, language-agnostic structure across several language pairs, while performance declines near the final layers.
- Experimental question: The experiment tests whether the mapping between translations depends only on the language pair, not on the sentence.Sentence representations are extracted from parallel sentences and compared across layers.
- Experimental setup: 5000 parallel sentence pairs are encoded separately, averaged into layer-specific sentence vectors, and used to estimate an average translation-direction vector.The procedure excludes [CLS] and [SEP] tokens before averaging representations.
- Evaluation: Nearest neighbor accuracy measures how often translating a sentence vector retrieves its correct paired sentence vector in the target language.The metric is computed using the closest target-language sentence vector.
- Results: Over 50% accuracy is achieved for EN-DE in all but the bottom layers, with similar curves for EN-RU and UR-HI.These results suggest that hidden representations share a common subspace containing language-agnostic linguistic information.
- Results: Accuracy declines in the final layers, possibly because language-model prediction requires more language-specific information.The proposed explanation concerns the model’s pre-training objective and its need to predict missing words.
6 Conclusion
The paper concludes that M-BERT develops multilingual representations without explicit multilingual training, but its transfer remains weaker for typologically divergent or transliterated targets. The authors suggest explicit multilingual objectives as a direction for addressing these deficiencies.
- Conclusion: M-BERT’s cross-lingual generalization is underpinned by multilingual representations learned without explicit multilingual training.The conclusion covers transfer across scripts and to code-switching.
- Conclusion: Transfer is effective across scripts and to code-switching, but less effective for typologically divergent and transliterated targets.The conclusion identifies systematic deficiencies affecting particular language-pair settings.
- Interpretation: Shared word pieces appearing across languages may encourage co-occurring pieces to occupy a shared representational space.The authors present this as a hypothesis for how multilingual structure emerges from monolingual training.
- Implications: The probing experiments are intended to focus future research on shortcomings of current contextualized word-representation approaches.The paper frames this as a direction for further inquiry rather than a demonstrated intervention.
A Model Parameters
The experiments use the BERT-Base Multilingual Cased checkpoint with fixed fine-tuning settings and no parameter tuning.
- Training settings: All models were fine-tuned for 3 epochs with batch size 32 and maximum sequence length 128.These settings were applied uniformly across the reported experiments.
- Training settings: Fine-tuning used learning rate 3e−5, 10% warmup, linear decay, and 10% dropout on the last layer.No parameter tuning was performed.
- Model: The experiments used the BERT-Base, Multilingual Cased checkpoint.The passage identifies the checkpoint as the released Google Research BERT model.
B CoNLL Results for EN-BERT
The EN-BERT comparison reports a substantial zero-shot performance gap relative to M-BERT, indicating that multilingual pre-training helps cross-lingual transfer.
- Comparison: EN-BERT has a large zero-shot performance gap compared with M-BERT on the CoNLL test sets.Rows denote fine-tuning languages, while columns denote evaluation languages.
C Some POS Results for EN-BERT
Table 8 reports POS accuracy for EN-BERT across selected European-language UD test sets. Its zero-shot performance falls well below M-BERT’s, indicating that pretraining helps learn useful cross-lingual grammatical representations.
- Table 8 reports POS accuracy on UD test sets for a subset of European languages using EN-BERT.Rows identify the fine-tuning language, while columns identify the evaluation language.
- EN-BERT’s zero-shot performance shows a big gap relative to M-BERT’s performance.
- The comparison indicates that pretraining helps learn a useful cross-lingual representation for grammar.