PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, Dietrich Klakow

arXiv:2204.06487v3cs.CL

TL;DR

Languages unseen during multilingual pre-training, especially African languages, show a performance gap, while language-specific adaptation requires separate specialized models. The paper introduces MAFT to adapt one model across 20 languages, combines it with vocabulary reduction, and finds competitive performance with smaller models and improved zero-shot transfer for parameter-efficient methods.

  • Problem

    Multilingual PLMs show a large performance gap on languages unseen during pre-training, while adapting each target language individually requires substantial disk space and limits cross-lingual transfer.

  • Method

    MAFT adapts multilingual PLMs simultaneously to 17 African languages and Arabic, French, and English, while removing non-African-script vocabulary tokens before adaptation.

  • Results

    MAFT is competitive with individual-language LAFT across two PLMs and three NLP tasks, while vocabulary reduction and MAFT reduce XLM-R parameter size by 50%.

  • Takeaways & Limitations

    A single adapted model can support cross-lingual transfer across many African languages, and it improves zero-shot transfer for parameter-efficient fine-tuning methods.

  • Takeaways & Limitations

    Vocabulary selection has no single sweet spot covering all languages, and the frequency-based assumption may fail for distant low-resource languages or domain-mismatched corpora.

Abstract

from arXiv · show

Multilingual pre-trained language models (PLMs) have demonstrated impressive performance on several downstream tasks for both high-resourced and low-resourced languages. However, there is still a large performance drop for languages unseen during pre-training, especially African languages. One of the most effective approaches to adapt to a new language is \textit{language adaptive fine-tuning} (LAFT) -- fine-tuning a multilingual PLM on monolingual texts of a language using the pre-training objective. However, adapting to a target language individually takes a large disk space and limits the cross-lingual transfer abilities of the resulting models because they have been specialized for a single language. In this paper, we perform \textit{multilingual adaptive fine-tuning} on 17 most-resourced African languages and three other high-resource languages widely spoken on the African continent to encourage cross-lingual transfer learning. To further specialize the multilingual PLM, we removed vocabulary tokens from the embedding layer that corresponds to non-African writing scripts before MAFT, thus reducing the model size by around 50%. Our evaluation on two multilingual PLMs (AfriBERTa and XLM-R) and three NLP tasks (NER, news topic classification, and sentiment classification) shows that our approach is competitive to applying LAFT on individual languages while requiring significantly less disk space. Additionally, we show that our adapted PLM also improves the zero-shot cross-lingual transfer abilities of parameter efficient fine-tuning methods.

1 Introduction

Multilingual PLMs remain weaker on languages unseen during pre-training, while language-specific adaptation improves transfer but requires separate specialized models. MAFT adapts one model to many African languages, reduces vocabulary-related size, and remains competitive across evaluations.

  • Multilingual PLMs face a performance gap on languages unseen during pre-training, particularly amid limited coverage of low-resource languages.
  • LAFT adapts a multilingual PLM to monolingual target-language text using the pre-training objective and improves cross-lingual transfer.Applying LAFT separately to each language requires substantial disk space and specializes models to individual languages.
  • MAFT adapts one multilingual PLM simultaneously to 17 African languages and three high-resource languages widely spoken in Africa.The approach is designed to provide a single model for cross-lingual transfer learning across these languages.
  • 50% model-size reduction results from removing embedding vocabulary tokens associated with non-Latin and non-Ge’ez scripts before MAFT.
  • Across AfriBERTa, XLM-R, and three NLP tasks, MAFT is competitive with language-specific LAFT while using one model instead of separate models.The adapted PLM also improves zero-shot cross-lingual transfer for parameter-efficient fine-tuning methods.
  • ANTC adds news-topic evaluation data for Lingala, Somali, Naija, Malagasy, and isiZulu, alongside publicly released code and data.

2 Related Work

Prior work expands African-language coverage through regional pre-training, language adaptation, and evaluation resources, but these approaches involve resource, specialization, or coverage constraints. This paper extends multilingual adaptation and contributes additional news-topic datasets.

  • Multilingual PLMs for African languages: Regional multilingual PLMs such as AfriBERTa address African-language under-representation, although AfriBERTa lacks several southern-Bantu languages.AfriBERTa has 126M parameters and was trained from scratch on 11 African languages.
  • Multilingual PLMs for African languages: MAFT extends adaptation to multiple African languages while preserving downstream performance on high-resource and low-resource languages.
  • Adaptation of multilingual PLMs: Existing adaptation methods modify multilingual models or extend their language coverage through new objectives, vocabularies, or training data.
  • Compressing PLMs: Distillation can create smaller multilingual PLMs but often reduces performance relative to the teacher model.
  • Evaluation datasets for African languages: African-language evaluation resources remain limited, with broad benchmarks concentrated in NER and sentiment and news-topic datasets covering fewer languages.
  • Evaluation datasets for African languages: ANTC contributes news-topic datasets for five additional African languages: Lingala, Naija, Somali, isiZulu, and Malagasy.

3 Data

The study adapts multilingual PLMs using monolingual news text from 20 languages and evaluates them on NER, news-topic classification, and sentiment classification. It also creates ANTC for five African languages.

  • MAFT covers 17 African languages and Arabic, French, and English, selected based on relatively large available monolingual text collections.
  • Monolingual adaptation data comes from mT5 News, Voice of America News, and other African news websites.
  • The experiments include sentence-level news-topic and sentiment classification plus token-level NER across English and diverse African languages.
  • NER evaluation uses MasakhaNER for 10 African languages, CoNLL 2003 for English, and additional datasets for isiXhosa and Arabic.
  • News-topic classification uses existing datasets spanning Amharic, English, Kinyarwanda, Kiswahili, Yorùbá, and Hausa.
  • Sentiment evaluation uses NaijaSenti for four Nigerian languages plus Amharic and English Twitter sentiment datasets.Only positive, negative, and neutral tweets are used.
  • ANTC is created from VOA, BBC, Global Voices, and Isolezwe news sources for Lingala, Somali, Naija, Malagasy, and isiZulu.

4 Pre-trained Language Models

The study compares multilingual PLMs differing in language coverage and parameter size, primarily adapting XLM-R-base. Evaluation uses three downstream tasks with models ranging from 117M to 550M parameters.

  • The evaluated multilingual PLMs use a masked-language-model objective and differ in parameter size and African-language coverage.
  • XLM-R covers 100 languages, including eight African languages, with XLM-R-base and XLM-R-large containing 270M and 550M parameters.
  • The main experiments use XLM-R-base.
  • AfriBERTa is a 126M-parameter model pre-trained only on African languages and previously shown competitive with XLM-R-base on African-language datasets.
  • XLM-R-miniLM is a distilled XLM-R-large model with 117M parameters.
  • The models are evaluated on NER, news-topic classification, and sentiment classification using task-specific fine-tuning settings.

5 Multilingual Adaptive Fine-tuning

The paper introduces MAFT to adapt multilingual masked PLMs simultaneously to African and other widely spoken African-continent languages, while reducing vocabulary size for a smaller model. Across NER, news topic classification, and sentiment classification, MAFT is often competitive with language-specific LAFT while using one model for many languages.

  • 5.1 Vocabulary reduction: Vocabulary reduction before MAFT produces AfroXLMR-small by removing tokens associated with non-African scripts.The authors compare removing tokens before versus after MAFT and report that removal before MAFT works better.
  • 5.1 Vocabulary reduction: k = 52,000 Amharic sub-tokens cover 99.8% of Amharic texts, while k = 60,000 tokens cover 99.6% of the other 19 languages.The two selected vocabularies are merged, with the top 1,000 original XLM-R tokens additionally included.
  • Performance on languages seen during pre-training: LAFT improves performance across nearly all evaluated languages and tasks, including +7.4 F1 on Amharic NER and +9.5 F1 on Zulu news-topic classification.English is the exception because XLM-R had already seen large amounts of English during pre-training.
  • 5.2.2 Multilingual adaptive fine-tuning results: MAFT is often competitive with LAFT, with average differences of −0.5 F1 for NER, −0.2 F1 for news topic classification, and −0.3 F1 for sentiment classification.Unlike LAFT, MAFT yields one adapted model applicable to many languages rather than a separate model for each language.
  • 5.2.2 Multilingual adaptive fine-tuning results: +2.6 F1 on NER, +3.6 F1 on news topic classification, and +1.5 F1 on sentiment classification are the average improvements reported for MAFT on XLM-R-base.The authors report that all PLMs improve after MAFT, with the largest gains for XLM-R-miniLM; AfriBERTa generally improves by less than 0.6 F1.

More efficient models using vocabulary reduction

Vocabulary reduction cuts model size by more than 50% before MAFT, but performance declines slightly as tokens are removed, especially for non-Latin-script languages.

  • More than 50% model-size reduction is achieved before applying MAFT through vocabulary reduction.
  • −1.6, −1.5 and −0.6 F1 are the average reductions for NER, news topic, and sentiment classification versus XLM-R-base+LAFT.
  • Vocabulary-reduced XLM-R-base-v70k remains better than similarly sized XLM-R-miniLM and unadapted PLMs, with or without MAFT.
  • Performance drops most for Amharic and Arabic because vocabulary reduction lowers tokenizer coverage of their Ge’ez and Arabic-script subwords.
  • Increasing Amharic sub-tokens reduces unknown tokens and improves F1, but decreases vocabulary coverage for other languages, leaving no universal selection strategy.
  • AfroXLMR-large is competitive with or better than individual language LAFT models while using a single model for downstream transfer.

6 Cross-lingual Transfer Learning

The cross-lingual transfer experiments compare domain-matched and Wikipedia-trained parameter-efficient adaptations on MasakhaNER, alongside fully supervised baselines. News-domain training improves transfer, with LT-SFT on AfroXLMR-base achieving the strongest overall performance.

  • The experiments evaluate zero-shot cross-lingual transfer using parameter-efficient fine-tuning methods on African-language NER.
  • The MasakhaNER comparison uses three tags, while its average excludes Amharic; fully supervised baselines measure the gap when annotated target-language data exists.
  • MAD-X 2.0 and LT-SFT use modular or sparse language-specific adaptations, while MAD-X 2.0 additionally supports modular language and task representations.
  • News-domain language adapters outperform largely Wikipedia-trained adapters by +4.2 for MAD-X and +3.7 for LT-SFT.
  • LT-SFT on XLM-R-base outperforms mBERT on all languages, with a few exceptions for MAD-X involving Hausa, Papiamento Creole, and Yoruba.
  • LT-SFT on AfroXLMR-base achieves the best overall performance and sometimes exceeds fully supervised results, including for Kinyarwanda, Luganda, Wolof, and Yoruba.

7 Conclusion

The paper proposes MAFT to adapt multilingual PLMs to many African languages with one model and evaluates it across three downstream tasks. MAFT remains competitive with LAFT while vocabulary reduction cuts XLM-R parameter size by 50%.

  • MAFT adapts multilingual PLMs to many African languages with a single model and is evaluated on three NLP downstream tasks.
  • A new news topic classification dataset is contributed for four African languages.
  • MAFT is competitive with LAFT while replacing many individually specialized models with one model.
  • 50% parameter-size reduction results from combining vocabulary reduction with MAFT while remaining competitive with individual-language LAFT.

A.1 Monolingual corpora for LAFT and MAFT

The LAFT and MAFT models use different monolingual corpora depending on language availability, with mC4 or VOA filling gaps in existing MasakhaNER resources.

  • MAFT models are trained on an aggregation of monolingual data listed in Table 9.
  • Existing XLM-R-base+LAFT models are used when available, while mC4 supplies most other languages and VOA supplies English.
  • The same monolingual corpus is used for XLM-R-base+LAFT and XLM-R-large+LAFT comparisons.

A.2 News corpora for language adapters and SFTs

The paper provides monolingual news corpora for training language adapters and supervised fine-tuning models in cross-lingual settings, documenting their sources, sizes, and sentence counts.

  • Table 10 provides the news corpus used to train language adapters and SFTs for cross-lingual settings.
  • Table 9 describes pre-processed monolingual corpora prepared following AfriBERTa’s approach.
  • The corpus documentation records source, size in megabytes, and number of sentences for the monolingual datasets.
Loading 2204.06487v3…