Source-linked AI summary
Extending Multilingual BERT to Low-Resource Languages
Zihan Wang, Karthikeyan K, Stephen Mayhew, Dan Roth
TL;DR
M-BERT’s cross-lingual success is limited to the 104 languages seen during pretraining, leaving many low-resource languages without coverage. The paper proposes Extend, which expands M-BERT for a target language and continues monolingual pretraining; NER experiments report improvements for both existing and new M-BERT languages. The approach is efficient, but it currently handles one language at a time and can reduce performance on non-target languages.
Problem
M-BERT covers only the top 104 Wikipedia languages, while many low-resource languages lack readily available supervised training data.
Method
Extend enlarges M-BERT’s vocabulary and model components for a target language, then continues pre-training on monolingual target-language data.
Results
Extend improves over M-BERT for languages both inside and outside its original coverage in experiments spanning 27 NER languages.
Takeaways & Limitations
Extend offers an efficient way to adapt M-BERT to missing languages while retaining benefits from its multilingual base.
Takeaways & Limitations
Extend handles one language at a time and can degrade performance on non-target languages.
Abstract
from arXiv · showhide
Multilingual BERT (M-BERT) has been a huge success in both supervised and zero-shot cross-lingual transfer learning. However, this success has focused only on the top 104 languages in Wikipedia that it was trained on. In this paper, we propose a simple but effective approach to extend M-BERT (E-BERT) so that it can benefit any new language, and show that our approach benefits languages that are already in M-BERT as well. We perform an extensive set of experiments with Named Entity Recognition (NER) on 27 languages, only 16 of which are in M-BERT, and show an average increase of about 6% F1 on languages that are already in M-BERT and 23% F1 increase on new languages.
1 Introduction
M-BERT supports cross-lingual transfer but excludes most of the world’s languages, limiting performance where supervised data is scarce. Extend adapts M-BERT to new languages efficiently, and experiments show gains on both covered and uncovered languages.
- Motivation: M-BERT covers only the top 104 Wikipedia languages, excluding languages such as Somali, Uyghur, Oromo, Hausa, Amharic, and Akan.These exclusions matter because supervised training data is particularly difficult to obtain for low-resource languages.
- Motivation: Training a new multilingual BERT from scratch is expensive, while B-BERT cannot use supervised data from multiple languages.The paper motivates a more efficient adaptation strategy than either alternative.
- Approach: Extend enlarges M-BERT’s vocabulary for a new language and continues pre-training on that language.The approach takes less than 7 hours to train with a single cloud TPU.
- Evaluation: Experiments evaluate zero-shot NER across 27 languages, including 11 languages absent from M-BERT.The study compares averaged performance for languages already in M-BERT and new languages, with supervised M-BERT performance as an upper bound.
- Results: Extend improves over M-BERT for languages both inside and outside its original 104-language coverage.The paper also reports that Extend is superior in most cases to training B-BERT from scratch.
2 Related works
The paper distinguishes Extend from prior cross-lingual approaches by adapting M-BERT to missing languages without requiring bilingual supervision. It attributes the method’s advantage over B-BERT to retaining M-BERT’s multilingual base.
- Prior approaches: Unlike BiCCA, LASER, XLM, and related approaches, Extend does not require bilingual dictionaries or parallel corpora.Such cross-lingual supervision is particularly challenging to obtain for low-resource languages.
- Comparison with continuing-training methods: Extend focuses on improving M-BERT for target languages by addressing missing word-pieces, whereas MonoTrans focuses on cross-lingual features from overlapping word-pieces.The comparison distinguishes the papers’ objectives rather than their training setup alone.
- Comparison with continuing-training methods: Extend outperforms B-BERT in several languages, while MonoTrans has performance similar to B-BERT.The authors connect Extend’s behavior to the multilinguality of M-BERT compared with BERT.
3 Background
M-BERT is a Wikipedia-pretrained bidirectional Transformer covering 104 languages and exhibiting cross-lingual transfer without explicit aligned data. B-BERT uses the same general training approach but contains only English and one target language.
- Multilingual BERT: M-BERT is a bidirectional Transformer language model pretrained on Wikipedia text from the top 104 languages.The languages are selected based on having the most Wikipedia articles.
- Multilingual BERT: M-BERT uses masked language modeling and next sentence prediction, then transfers across languages after fine-tuning on high-resource supervised data.The target language is evaluated after fine-tuning, typically on English data.
- Bilingual BERT: B-BERT follows M-BERT’s training procedure but contains only English and the target language.Prior work has demonstrated cross-lingual effectiveness for both M-BERT and B-BERT on NER and other tasks.
4 Our Method: Extend
Extend adapts M-BERT to a target language by expanding its vocabulary and model parameters, then continuing monolingual pre-training. Existing M-BERT weights are retained while newly added parameters are initialized separately.
- Training protocol: Extend expands the vocabulary, encoder, and decoder to accommodate the target language before continued pre-training.The continued pre-training uses monolingual data from that target language.
- Vocabulary extension: The new vocabulary is built from target-language monolingual data after filtering words already present in M-BERT.The paper sets the new vocabulary size to Vnew = 30000.
- Parameter extension: The encoder and decoder receive appended parameters for the new vocabulary, while the decoder shares weights with the encoder apart from an additional bias.The extended encoder has dimensions (Vmbert + Vnew) × d.
- Initialization: All original parameters retain M-BERT’s pretrained weights, while the newly appended encoder and decoder parts are initialized separately.The resulting trained model is called E-MBERT.
5 Experiments
Experiments evaluate Extend on zero-shot cross-lingual NER across languages already covered and absent from M-BERT, comparing it with M-BERT and B-BERT under several data and training settings. Extend generally improves performance, converges faster than B-BERT, and can retain gains without additional monolingual data.
- Experimental settings: The experiments use LORELEI text and NER data, evaluate zero-shot NER with F1, and average scores across five random seeds.Extend is trained for 500k iterations, while B-BERT is trained for 2M iterations.
- E-MBERT versus M-BERT: In almost all languages, E-MBERT outperforms M-BERT regardless of whether the language is present in M-BERT.The comparison uses English supervision for zero-shot cross-lingual NER; supervised target-language M-BERT provides an upper-bound benchmark.
- E-MBERT versus M-BERT: E-MBERT improves over M-BERT even for languages already included in M-BERT.The paper attributes this to expanded target-language vocabulary, greater target-language focus during continued pre-training, and additional monolingual data.
- Extend without extra data: E-MBERT performance does not degrade when Extend uses Wikipedia data rather than additional monolingual data.This comparison examines whether E-MBERT’s effectiveness depends on extra training data.
- E-MBERT versus B-BERT: E-MBERT often outperforms B-BERT on languages absent from M-BERT while using the same target-language corpus.B-BERT is trained for 2M steps, whereas E-MBERT requires 500k steps in the reported protocol.
- Convergence: E-MBERT converges within 100k steps, whereas B-BERT takes more than 1M steps on the evaluated Hindi and Sinhala settings.The authors characterize E-MBERT as much more efficient than B-BERT.
- Non-target languages: Extending M-BERT toward a target language degrades performance on other non-target languages.The paper reports evaluations of Hindi and Sinhala E-MBERT models on other languages in Appendix A.2.
6 Conclusions and Future work
The paper concludes that Extend improves M-BERT for languages both inside and outside its original coverage and compares favorably with B-BERT. Future work includes extending multiple languages simultaneously, initializing new embeddings with alignment models, and applying the approach to multilingual RoBERTa.
- Conclusions: Extend improves performance across languages compared with both M-BERT and B-BERT.The conclusion presents Extend as a method for languages not included in M-BERT while also reporting benefits over the baselines.
- Future work: Extend currently handles one language at a time, motivating future work on extending multiple languages simultaneously.This is identified as a future direction rather than a reported capability of the current method.
- Future work: Future work could initialize new vocabulary embeddings with MUSE or VecMap alignment models and bilingual dictionaries instead of random initialization.The proposal concerns initialization of embeddings for the added vocabulary.
- Future work: The approach could also be applied to stronger multilingual models such as RoBERTa.The paper frames this as a possible extension of the method.
A.1 Performance of E-MBERT on English:
E-MBERT preserves English-source performance relative to M-BERT in nearly all evaluated cases.
- English-source performance: E-MBERT’s English performance is almost as good as M-BERT’s, except for a few languages.The paper states that knowledge of English, the source language, is not affected.
A.2 Detailed data on all languages
Extend improves both cross-lingual and supervised performance in nearly all evaluated cases, but extending one language can reduce transferability to other languages.
- Extend is useful for both cross-lingual and supervised performance in almost all cases.
- Extending on one language hurts transferability to other languages.
A.3 Comparison between B-BERT and E-MBERT:
The comparison evaluates E-MBERT and B-BERT across English and target languages, using M-BERT as an English reference. E-MBERT generally performs better on target languages, while B-BERT is mostly better on English.
- B-BERT is mostly better than Extend on English, whereas Extend performs better on the target language in most cases.
- Table 3 reports English NER performance for M-BERT and E-MBERT.
- Table 4 orders columns by supervised and zero-shot performance for M-BERT and E-MBERT, followed by Hindi and Sinhala evaluations and dataset statistics.
- E-MBERT is compared with B-BERT on both English and target-language performance, with M-BERT included as an English reference.