PaperlayerView exact revision on arXiv

Source-linked AI summary

MasakhaNER: Named Entity Recognition for African Languages

David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, Salomey Osei

arXiv:2103.11811v2cs.CLcs.AI

TL;DR

African languages are underrepresented in NLP datasets and NER resources. The paper creates high-quality NER datasets for ten African languages, evaluates baseline and transfer-learning models, and releases resources; language-specific adaptation improves F1 by 1–7% over corresponding multilingual baselines.

  • Problem

    African languages are scarcely represented in NLP datasets, research, and tools, including NER resources needed for reproducible evaluation and models suited to their challenges.

  • Method

    The authors develop locally sourced, native-speaker-annotated NER datasets for ten African languages and evaluate CNNBiLSTMCRF, mBERT, XLMR, and transfer-learning approaches.

  • Results

    1–7% improvement in F1score is achieved by language-specific BERT and XLMR models over fine-tuned mBERTbase and XLMRbase, respectively.

  • Takeaways & Limitations

    The released datasets, code, and models provide resources for studying African-language NER, transfer across languages, and open challenges.

  • Takeaways & Limitations

    The authors plan to increase annotated sentences per language and expand the dataset to more African languages.

Abstract

from arXiv · show

We take a step towards addressing the under-representation of the African continent in NLP research by creating the first large publicly available high-quality dataset for named entity recognition (NER) in ten African languages, bringing together a variety of stakeholders. We detail characteristics of the languages to help researchers understand the challenges that these languages pose for NER. We analyze our datasets and conduct an extensive empirical evaluation of state-of-the-art methods across both supervised and transfer learning settings. We release the data, code, and models in order to inspire future research on African NLP.

1 Introduction

The paper addresses the underrepresentation of African languages in NLP by developing NER resources for ten widely spoken languages with local stakeholder involvement. It evaluates models and releases datasets, code, and models to support future research.

  • African languages are underrepresented in NER because datasets, reproducible results, and researchers familiar with their challenges are limited.
  • The authors bring together language speakers, dataset curators, NLP practitioners, and evaluation experts to address African-language NER challenges.
  • The study develops NER datasets, models, and evaluation covering ten widely spoken African languages using online news corpora and language annotators.
  • The datasets are curated from local sources to ensure relevance for native speakers of the respective languages.
  • The experiments provide insights into transfer across languages and highlight open challenges.
  • The authors release the datasets, code, and models to facilitate future research on NER challenges in African languages.

2 Related Work

Prior African-language NER resources include broad but silver-standard WikiAnn coverage, smaller specialized corpora, and non-public language packs. Existing NER research also includes neural and multilingual pretrained-language-model approaches.

  • Most existing NER datasets are in high-resource languages, while WikiAnn covers 282 languages using silver-standard labels transferred from English through knowledge-base links.
  • Most African languages in WikiAnn have fewer than 10k tokens, limiting the size of their available resources.
  • Other African-language resources include SADiLaR for ten South African languages, small Yorùbá and Hausa corpora, and non-public LORELEI language packs.
  • Established NER models include CRF, CNNBiLSTM, BiLSTMCRF, and CNNBiLSTMCRF, while neural systems commonly use pretrained word embeddings.
  • Multilingual pretrained models such as mBERT and XLM-RoBERTa enable NER transfer learning across several languages.

3 Focus Languages

The study covers ten widely spoken African languages selected for available news corpora and annotators, while emphasizing substantial script, phonological, orthographic, and named-entity variation that challenges English-oriented NER transfer.

  • Language selection and coverage: The ten languages were selected for online news availability, annotators, and status as widely spoken native African languages across East and West Africa.The authors organize them by language family and region to assess possible linguistic or entity-overlap proximity.
  • Named-entity variation: Named entities may share forms across languages while also reflecting regional usage and localized spellings.For example, “Zhang” is shared across Latin-script languages, whereas “Kano” receives language-specific accents.
  • Language characteristics: The languages differ substantially in writing systems, including Amharic’s Fidel script, Latin-based alphabets, additional consonants, and language-specific diacritics.Amharic has more than 231 Fidel characters, while several languages use modified Latin orthographies.
  • Language characteristics: The languages also vary in phonology, tone, morphology, and orthographic conventions, including tonal contrasts, agglutination, and an evolving NigerianPidgin orthography.These properties illustrate why surface forms and tokenization can differ across languages.
  • Named-entity variation: English-based NER systems face lexical, script, diacritic, tokenization, and spelling differences across the ten languages.Examples include no lexical overlap for Amharic, different spellings of “Nigeria,” and written-out numbers in Igbo, Wolof, and Yorùbá.

4 Data and Annotation Methodology

The authors build news-based NER datasets for ten African languages using native regional annotators, four entity categories, and collaborative procedures designed to achieve high agreement.

  • Data sources and annotators: The datasets use local news sources and were annotated with ELISA by native speakers from the same regions as the source material.Annotators volunteered through the Masakhane community and were trained with the MUC6 annotation guide.
  • Dataset statistics: Table 3 reports each dataset’s source, split sizes, annotator count, entity-label counts, named-entity token percentage, and total token count.These statistics characterize the composition and scale of the ten-language corpus.
  • Annotation scheme: The annotation scheme contains four entity types: personal name, location, organization, and date and time.The authors replace the CoNLL-style MISC category with DATE because MISC may be ill-defined for nonexpert annotators.
  • Quality control: Collaborative language-specific workshops allowed annotators to discuss and resolve disagreements during dataset creation.ELISA also provided entity-level F1 feedback and an interface for correcting annotation mistakes.
  • Quality control: Interannotator agreement ranged from 0.96 to 1.0 by entity-level F1, with perfect Fleiss’ kappa reported for Kinyarwanda, Luo, Swahili, and Wolof.For those four languages, two annotators labeled each token and resolved conflicts through discussion.

5 Experimental Setup

The experiments compare neural and multilingual transformer NER baselines trained in-language, then extend XLMR with language adaptation, gazetteers, domain transfer, crosslingual transfer, and regional aggregation.

  • Baseline models: CNNBiLSTMCRF, mBERT, and XLMR are trained on each language’s in-language training data and evaluated on its test data.The neural and transformer baselines provide the main supervised comparison across the ten datasets.
  • Baseline models: CNNBiLSTMCRF combines CNN character encodings with word embeddings, while mBERT and XLMR are fine-tuned multilingual transformers.The CNNBiLSTMCRF model uses randomly initialized word embeddings because suitable pretrained embeddings are unavailable for all languages.
  • Baseline models: MeanEBiLSTM extracts mean sentence representations from mBERT or XLMR before applying a BiLSTM and linear classifier.The approach uses the mean of the transformer’s 12-layer output embeddings rather than the [CLS] representation.
  • Adaptation and transfer: Language-adaptive fine-tuning uses unlabeled corpora from JW300, indigenous news sources, and XLMR Common Crawl data before NER training.The procedure adapts pretrained transformer models to each African language or its available domain data.
  • Enhancement experiments: The enhancement experiments apply gazetteers, transfer learning, and regional language aggregation to XLMR, focusing on PER, ORG, and LOC.DATE is excluded because Wikipedia gazetteers and some transfer sources lack DATE annotations; XLMR is used because it generally outperforms mBERT.
  • Adaptation and transfer: The study evaluates cross-domain transfer from Wikipedia to news and crosslingual transfer from English and Swahili NER data to other African languages.WikiAnn supports Amharic, Igbo, Kinyarwanda, Swahili, and Yorùbá, while CoNLL2003 provides over 14K English training sentences in the same news domain.
  • Transfer evaluation: 37.15 F1 was obtained with CoNLL2003 in zero-shot transfer, compared with 19.38 F1 for the OntoNotes 5.0 setup.The OntoNotes labels were remapped by combining FAC with ORG, GPE with LOC, and other categories with O except PER.

6 Results

Experiments compare supervised baselines, pretrained multilingual models, gazetteer features, and transfer strategies across ten African languages. Performance depends on language support, script, domain and source-language proximity, while language-adaptive pretraining and high-coverage gazetteers generally improve results.

  • Baseline models: Greater OOV rates generally correspond to lower CNNBiLSTMCRF performance on the ten in-language test sets.
  • Baseline models: Fine-tuned XLMRbase performs significantly better on five languages, while mBERT performs better for Yorùbá.Pretrained models also achieve reasonable performance on several languages absent from their training data.
  • Baseline models: Script and language-family coverage shape transfer: mBERT scores 0.0 F1 on Amharic, while XLMRlarge generally outperforms XLMRbase and mBERT.mBERT remains slightly better for Igbo, Luo, and Yorùbá.
  • Baseline models: Language-adaptive BERT and XLMR improve F1 by 1–7% over their multilingual counterparts, including over 4% for Wolof and over 60% for Amharic BERT.The approach remains effective with small, good-quality pretraining corpora.
  • Gazetteer features: Gazetteer features improve average performance, with larger gazetteers producing greater gains than smaller ones.Swahili uses 16K entities and NigerianPidgin uses a 2M-entity English gazetteer, whereas Amharic and Luganda use 2K and 500 entities.
  • Cross-domain transfer: WikiAnn-to-news zero-shot transfer stays below 40 F1 for every language, and in-domain fine-tuning does not improve over XLMRbase.Kinyarwanda and Yorùbá remain below 10 F1 in zero-shot transfer.
  • Cross-lingual transfer: Hausa and Swahili provide the strongest zero-shot cross-lingual transfer, with Hausa improving over English CoNLL and Swahili by over 16% and 1%, respectively.The pcm model, trained on only 2K sentences, transfers similarly to engCoNLL03 trained on 14K sentences.
  • Multilingual training: Training on all nine languages improves performance for every language except Swahili, averaging 1.6 F1 over six languages.

7 Conclusion and Future Work

The paper contributes high-quality NER resources and evaluations for ten African languages, including released models and transfer-learning analyses. Future work targets stronger baseline initialization, more annotations, and broader language coverage.

  • The authors create a high-quality NER dataset for ten African languages by bringing together multiple stakeholders.
  • They evaluate state-of-the-art NER models, establish baselines, release a multilingual model, and study cross-domain and cross-lingual transfer.
  • Future work will use pretrained word embeddings, increase annotated sentences per language, and expand the dataset to more African languages.

A.1 Annotator Agreement

Annotator agreement is high across the ten-language datasets, with perfect agreement reported for four languages. The most common disagreement concerns organization versus location labels.

  • The most common entity-level annotator disagreement is between organizations and locations.

A.2 Model Hyperparameters for Reproducibility

The experiments use fixed transformer and BiLSTM configurations to support reproducibility. All experiments run on a single Nvidia V100 GPU.

  • Transformer models: mBERT and XLMR fine-tuning uses 50 epochs, batch size 32, and learning rate 5e5, with maximum sequence lengths of 164 and 200 respectively.
  • All experiments were performed on a single Nvidia V100 GPU.

A.3 Monolingual Corpora for Language Adaptive Finetuning

The study uses monolingual corpora for language-adaptive finetuning, primarily drawing from JW300 and CC100, with CCAligned used for some languages. It also describes corpus preprocessing and finetuning implementation choices.

  • JW300 and CC100 provide monolingual data for most languages.
  • CCAligned is used for some languages, with duplicated sentences removed from CC100 when applicable.
  • Language-model finetuning uses HuggingFace code with a reported learning rate of 5e5.
Loading 2103.11811v2…