PaperlayerView exact revision on arXiv

Source-linked AI summary

When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models

Benjamin Muller, Antonis Anastasopoulos, Benoît Sagot, Djamé Seddah

arXiv:2010.12858v2cs.CL

TL;DR

The paper asks how language-model transfer should serve unseen languages with limited data and no multilingual-model coverage. It compares multilingual and monolingual models across 15 languages and finds diverse transfer behaviors, with script-related transliteration substantially improving performance for Hard languages.

  • Problem

    Many low-resource languages lack language-model coverage, while unseen languages differ in script, data availability, and relation to pretraining languages.

  • Method

    The study compares multilingual and monolingual models and adapts multilingual models through masked-language-model tuning on unseen-language raw data across three NLP tasks.

  • Results

    The 15 unseen languages exhibit Easy, Intermediate, and Hard behaviors, with script-related transliteration producing large downstream gains for Hard languages.

  • Takeaways & Limitations

    Transliterating Hard languages into a script used by a related pretrained language can make multilingual models more useful for those languages.

  • Takeaways & Limitations

    The categorization is based on relative mBERT performance against strong non-contextual baselines, leaving absolute performance analysis for future work.

Abstract

from arXiv · show

Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. Transliterating those languages improves very significantly the ability of large-scale multilingual language models on downstream tasks.

1 Introduction

The paper investigates how multilingual and monolingual language models transfer to unseen low-resource languages. It identifies distinct transfer behaviors and finds that transliteration can substantially improve multilingual-model performance for Hard languages.

  • Motivation: Most spoken languages, including Sorani Kurdish and Bambara, are not covered by available large-scale language models.The paper notes that even languages with millions of speakers remain uncovered.
  • Motivation: Training ever larger multilingual models is impractical, and multilingual models can be sub-optimal for under-sampled languages.The paper attributes the scaling challenge to the curse of multilinguality.
  • Approach: The study evaluates language-model and task adaptation on 15 typologically diverse languages across POS tagging, dependency parsing, and named-entity recognition.These experiments target usable representations for under-studied low-resource languages.
  • Findings: The authors classify unseen-language behavior into three categories: Easy, Intermediate, and Hard.The categories reflect how pretrained multilingual models can be used for low-resource languages.
  • Findings: Hard languages are predominantly spoken by communities underserved by modern NLP, and their transfer difficulty is linked to the writing script.The paper focuses on these languages because transliteration yields large gains over non-contextual strong baselines.
  • Contribution: The paper proposes transliteration into a better-handled script as a way to make multilingual models useful for additional unseen languages.The contribution specifically highlights Hard languages and typically uses Latin or a script associated with a related pretrained language.

2 Background and Motivation

Many languages lack annotated resources and language-model coverage, making contextualized modeling difficult. The paper examines how script, data availability, and linguistic relation shape transfer, emphasizing transliteration for languages that lag behind.

  • Motivation: Most of the world’s 6500+ languages have few or no annotated datasets, creating a major divergence in NLP technology coverage.The paper connects broader language-model development with inclusion, endangered-language preservation, and tailored NLP ecosystems.
  • Existing Approaches: Multilingual representation models have enabled transfer from high-resource to low-resource languages and improved downstream task performance.Recent models use sub-word tokenization and operate with an open vocabulary.
  • Existing Approaches: Low-resource languages can be addressed by training contextualized models on available data or adapting multilingual pretrained models.The background discusses both monolingual pretraining and unsupervised multilingual-model adaptation.
  • Coverage Gap: Approximately 120 languages have monolingual or multilingual language-model coverage, leaving many languages—including Bambara, Sorani Kurdish, and Maltese—without access to this technology.The coverage gap includes languages spoken by millions and an official European Union language.
  • Research Question: Across 15 typologically diverse unseen languages, behavior varies with script, available data, and relation to pretraining languages.The study focuses on languages that lag behind easier-to-handle counterparts to examine the role of script.
  • Research Question: Transliterating an unseen language into a script used by a related pretrained language can significantly improve downstream performance.This provides the background’s central proposed direction for difficult transfer cases.

3 Experimental Setting

The experiments compare baselines, monolingual models, and multilingual models on 15 unseen languages and three downstream tasks. Multilingual models are also adapted with masked-language-model tuning on target-language raw data.

  • Languages: The study defines unseen languages as languages not covered by pretrained language models and selects 15 typologically diverse languages absent from mBERT pretraining.The selection spans multiple language families and scripts and requires evaluation data for at least one downstream task.
  • Raw Data: Monolingual pretraining and fine-tuning use deduplicated OSCAR datasets, with Wikipedia or other collected data substituted for several languages.The substitutions apply where OSCAR lacks sufficient data.
  • Baselines: UDPipe future provides the POS-tagging and dependency-parsing baseline, while an LSTM-CRF model provides the named-entity-recognition baseline.The baselines use pretrained static word embeddings and character-level representations.
  • Language Models: MLM-TUNING adapts a multilingual model by fine-tuning its masked-language-model objective directly on raw data from the unseen target language.The resulting adapted mBERT model is called mBERT+MLM.
  • Models and Tasks: The experiments compare mBERT, mBERT+MLM, monolingual MLM, and strong non-contextual baselines on POS, parsing, and NER.Table 1 reports these comparisons for Easy Languages.
  • Dataset Splits: For datasets with fewer than 500 training sentences, the study concatenates training and test data and uses 8-fold cross-validation.This ensures at least 500 training sentences in nearly all experiments and treats the setting as few-shot learning.

4 The Three Categories of Unseen Languages

The paper identifies three behavioral categories among unseen languages: Easy languages benefit from mBERT directly, Intermediate languages require MLM-TUNING, and Hard languages remain difficult even after adaptation. The authors examine language-specific patterns and find that script differences are especially important for Hard languages.

  • Method: The experiments compare training language models from scratch, direct TASK-TUNING of mBERT, and MLM-TUNING followed by supervised fine-tuning against strong non-contextual baselines.The evaluation covers 15 typologically diverse languages and three NLP tasks: POS tagging, dependency parsing, and NER.
  • Easy: Easy languages achieve good performance from mBERT without extra MLM-TUNING and can outperform strong non-contextual baselines.Faroese, Swiss German, Naija, and Mingrelian are classified as Easy.
  • Intermediate: Intermediate languages generally trail strong non-contextual baselines with untuned mBERT, but MLM-TUNING enables usable models that can outperform those baselines.Examples include Livvi, Erzya, Maltese, Narabizi, Bambara, and Wolof; Wolof reaches 77.9 LAS after MLM-TUNING versus 77.0 for the non-contextual baseline.
  • Hard: Hard languages remain difficult because mBERT underperforms non-contextual and often monolingual models, while MLM-TUNING has minimal impact.For Uyghur, the non-contextual baseline beats contextual variants by up to 20 points in parsing, and UyghurBERT trained on 105K sentences outperforms mBERT after MLM-TUNING.
  • Scope: The categorization is based only on mBERT’s relative performance after fine-tuning against strong non-contextual baselines, not on absolute performance.The authors leave analysis of absolute performance and fine-tuning dataset-size effects for future work.
  • Script effects: The contrast between Livvi and Erzya, and the Uyghur results, suggests that script differences critically affect multilingual transfer, especially when related pretraining languages use other scripts.Uyghur uses Perso-Arabic script, whereas relevant Turkic pretraining languages use Latin or Cyrillic scripts.

5 Tackling Hard Languages with Multilingual Language Models

The paper tests whether transliteration helps mBERT transfer to unseen languages, finding that script compatibility can determine transfer success. Benefits vary substantially across languages and can include large downstream gains, but transliteration may also hurt performance.

  • Transfer via Transliteration: mBERT associates unseen languages with pretrained languages partly through script, making shared script a key condition for transfer.The authors hypothesize that script helps models correctly process unseen languages and explain Uyghur's association with Turkish after transliteration.
  • Transfer via Transliteration: The experiments compare mBERT adaptation and task tuning with monolingual models trained from scratch and non-contextual strong baselines.The evaluation covers six languages from four language families, with transliteration directed toward scripts represented in related-language pretraining data.
  • Transfer via Transliteration: Table 4 reports significant gains for Uyghur, Sorani, and Meadow Mari, minor effects for Erzya and Buryat, and no help for Mingrelian.Across these experiments, mBERT+MLM is reported as the best approach.
  • Transfer via Transliteration: The transliteration strategy maps source-script characters to frequent phoneme renderings in a related Latin-script target language.Target languages include Turkish for Uyghur and Kurmanji Kurdish for Sorani; other languages use standard transliteration modules.
  • Transfer via Transliteration: Transliteration improves Uyghur parsing by 16 points and NER by 20 points, while Meadow Mari gains 8 F1 points on NER.Uyghur is transliterated from Arabic to Latin, and Meadow Mari is among the Uralic languages tested.
  • Transfer via Transliteration: Transliteration is not universally beneficial: Mingrelian performance drops, and high-resource Arabic, Russian, and Japanese degrade when converted to Latin.For Arabic, the reported drops are 1.5 points in POS tagging, 4.1 in parsing, and 6.9 in NER.

6 Discussion and Conclusion

The discussion frames reuse of multilingual models as a practical alternative to training ever larger or language-specific models for under-resourced languages. The results indicate that script-related transfer limitations especially constrain Hard languages, while transliteration can improve performance when it connects them to related pretrained languages.

  • Discussion and Conclusion: Training one multilingual model for all languages is impractical because multilinguality and the Zipfian distribution of languages require ever larger models.The paper presents reuse of pretrained models as a more cost-efficient and environmentally reasonable alternative.
  • Discussion and Conclusion: Adapter layers offer parameter-efficient multilingual transfer, but show no significant supervised improvement over simpler masked-language-model fine-tuning.The paper also argues that a language-agnostic adaptation method is difficult to justify given human languages' typological diversity.
  • Discussion and Conclusion: The authors leave vocabulary adaptation on additional scripts, languages, and tasks for future work.They expect vocabulary adaptation to be orthogonal to transliteration but do not evaluate that combination here.
  • Discussion and Conclusion: The authors find that Hard languages remain outside the scope of available language models, with the challenge mostly attributable to script.Transliteration to a script used by a related higher-resource pretrained language produces large downstream improvements.
  • Discussion and Conclusion: Cross-script transfer is possible only when the model saw related languages in the same script during pretraining.This conclusion qualifies earlier suggestions that multilingual models can transfer efficiently across scripts in zero-shot settings.
  • Discussion and Conclusion: The proposed approach may help bootstrap NLP resources and tools for languages currently underserved by the NLP community.The authors connect this possibility to the emergence of NLP ecosystems for those languages.

A Languages

The paper's language appendix lists the 15 typologically diverse unseen languages used in the experiments and records their linguistic and dataset characteristics.

  • Languages: Table 12 lists the 15 unseen languages together with their language families, scripts, origins, and available sentence counts.The table also records the Easy, Intermediate, and Hard category assigned to each language.
  • Languages: The experimental data come primarily from Universal Dependencies and WikiNER, with the CoNLL-2003 English NER dataset also used.These sources support the paper's POS tagging, dependency parsing, and named-entity recognition experiments.

B Reproducibility

The reproducibility appendix documents computational resources, optimization, hyperparameter tables, and evaluation procedures for the paper's fine-tuning experiments.

  • Reproducibility: Experiments ran on a shared cluster equivalent to 15 Nvidia Tesla T4 GPUs.This reports the infrastructure used for the experimental runs.
  • Reproducibility: Fine-tuning uses Adam, selects hyperparameters by validation loss, and reports test scores averaged across five random-seed runs.The appendix directs readers to Tables 6–7 for hyperparameters and bounds.
  • Reproducibility: Table 6 records task-specific fine-tuning hyperparameters, grid-search trial counts, and average training-and-evaluation runtime in minutes.The table's best settings are selected on the validation set within stated bounds.
  • Reproducibility: Table 7 reports the hyperparameters used for unsupervised fine-tuning.This corresponds to the MLM-TUNING adaptation stage described in the experimental design.

C Easy Languages

Easy unseen languages can benefit strongly from multilingual transfer, especially when related pretraining languages and scripts are available. Task- or language-specific tuning further improves performance, including in zero-shot settings.

  • Easy languages: Easy languages can achieve very high zero-shot performance when task-tuning uses related languages.This category includes languages that are readily handled through transfer from related languages.
  • Faroese: Faroese reaches more than 96.5% POS-tagging accuracy, 86% LAS, and 58% on the reported downstream evaluation after mBERT fine-tuning.mBERT outperforms the monolingual FaroeseBERT baseline, and fine-tuning on Faroese text yields further improvements.
  • Swiss German: Swiss German gains more than 25 points in both POS tagging and dependency parsing after MLM fine-tuning with 200K sentences, even in zero-shot settings.Similar improvement trends appear in few-shot settings, and the multilingual model outperforms baselines in both settings.
  • Naija: Naija is classified as easy because the multilingual approach surpasses the monolingual baseline after language- and task-tuning on 6K training examples.Naija is an English creole also known as Nigerian English or Nigerian Pidgin.
  • Other English varieties: Early Modern English and Singlish are also treated as easy unseen languages because their English variants can be handled by mBERT.Both are technically unseen by mBERT but remain easy to adapt because they are English varieties.

D Additional Uralic languages experiments

Additional Uralic experiments place the languages in the Intermediate category: related-language task-tuning helps, while script similarity strongly affects zero-shot transfer.

  • Script effects: Livvi’s Latin script supports almost twice the zero-shot performance achieved by Uralic languages written in Cyrillic.Finnish and Estonian provide the task-tuning languages, and Livvi shares their Latin script.
  • Intermediate languages: Low-resource Uralic languages are Intermediate because small amounts of annotated data improve mBERT beyond its task-tuned performance baseline.They are not easy enough for zero-shot handling but benefit from pretrained multilingual representations and related-language transfer.
  • Results: For Livvi and Erzya, mBERT with MLM-TUNING achieves the best performance, exceeding the non-contextual baseline by more than 1.5 point for parsing and POS tagging.The result holds across both reported task types.

E Controlled experiment: Transliterating High-Resource Languages

A controlled experiment compares mBERT on original versus Latin-transliterated text for high-resource languages seen during pretraining. Transliterating Arabic, Japanese, and Russian to Latin produces large performance drops.

  • Experimental design: The experiment compares mBERT fine-tuned and evaluated on original-script text against mBERT fine-tuned and evaluated on transliterated text.The languages examined are Arabic, Japanese, and Russian, all seen during mBERT pretraining.
  • Results: Transliterating Arabic, Japanese, and Russian to the Latin script leads to large drops in mBERT performance.The comparison evaluates the effect of transliteration while keeping the model and evaluation setup otherwise aligned.
  • Data: Table 12 records the sentence counts used for training monolingual models from scratch and for MLM-TUNING mBERT.The table also marks code-mixed data with a French-related annotation.
Loading 2010.12858v2…