Source-linked AI summary
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation
David Ifeoluwa Adelani, Jesujoba Oluwadara Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Chinenye Emezue, Colin Leong, Michael Beukman, Shamsuddeen Hassan Muhammad, Guyo Dub Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Benjamin Ayoade Ajibade, Tunde Oluwaseyi Ajayi, Yvonne Wambui Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, Anuoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Koffi Kalipe, Derguene Mbaye, Allahsera Auguste Tapo, Victoire Memdjokam Koagne, Edwin Munkoh-Buabeng, Valencia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda, Andiswa Bukula, Sam Manthalu
TL;DR
Low-resource African languages are underrepresented in multilingual pre-training, limiting translation coverage and domain transfer. The paper creates a 16-language African news corpus and adapts large pre-trained models, finding that small amounts of high-quality translation data are highly effective. The work supports adapting translation systems to new languages and domains with limited parallel data.
Problem
Pre-trained multilingual models largely omit low-resource African languages, raising how they can be adapted to unseen languages and new domains.
Method
The authors create a 16-language African news corpus and adapt multilingual pre-trained models using high-quality parallel data, continued pre-training, and fine-tuning.
Results
Fine-tuning large pre-trained models on small quantities of high-quality translation data effectively adapts them to previously unseen African languages and additional domains.
Takeaways & Limitations
As little as 2k sentences can support new translation systems for languages spoken by millions but poorly represented on the web.
Takeaways & Limitations
The authors leave vocabulary changes and MT-focused word-alignment objectives for future work.
Abstract
from arXiv · showhide
Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pre-trained models be used for languages not included in the initial pre-training? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of high-quality translation data.
1 Introduction
The paper asks how pre-trained multilingual models can support translation for unseen African languages and transfer across domains. It introduces a 16-language study and finds that a few thousand high-quality translated sentences can effectively adapt these models.
- The work investigates translation for 16 underrepresented African languages with little or no available training data.
- African-language news translation addresses limited access to information otherwise published mainly in English, French, or Arabic.The paper highlights this bottleneck as especially critical during crises.
- Fine-tuning pre-trained models on a few thousand high-quality translation sentences is remarkably effective for unseen African languages and domains.The study compares zero-shot transfer, continual pre-training, and multilingual fine-tuning strategies.
- The authors create a new African news corpus and adapt multilingual pre-trained models including MT5, ByT5, mBART, and M2M-100.
- The study quantifies domain-transfer effects using small in-domain translation sets and compares alternative fine-tuning strategies.
2 Related Work
Related work spans African parallel-corpus construction, low-resource and unsupervised machine translation, and transfer from multilingual pre-trained models. Existing evidence favors transfer learning over baseline approaches, while unsupervised methods often perform poorly in low-resource settings.
- African-language parallel corpora are often small and low quality when automatically crawled, while cleaner sources commonly come from religious texts.
- Prior low-resource translation research includes unsupervised, multilingual, and related-language transfer methods.
- Unsupervised machine translation typically performs poorly for low-resource languages despite leveraging monolingual or related-language data.
- Transfer from multilingual pre-trained and large-scale multilingual translation models often outperforms bilingual and Transformer baseline models for low-resource languages.
3 Focus Languages and Their Data
The study covers 16 African languages spanning multiple language families and resource levels, all spoken by at least one million people. Available parallel data is concentrated in religious texts, with substantial linguistic variation across the languages.
- The 16 focus languages range from moderately to very low resource and span Afro-Asiatic, Nilo-Saharan, English Creole, and Niger-Congo families.Most languages belong to Niger-Congo, and all are spoken by at least one million people.
- The dataset design does not focus exclusively on English because French is central for many Francophone African communities.
- The languages differ substantially in script conventions, tone, diacritics, morphology, and word formation while sharing Subject-Verb-Object order.
- Existing parallel data is primarily religious, with JW300 covering 11 languages and Bible data supplying the remaining five.Bible alignments total around 31k verses, with smaller collections for Ghomálá’, Bambara, and Wolof.
4 MAFAND-MT African News Corpus
MAFAND-MT is a participatorily created African news corpus built from local sources, professional translation, and native-speaker quality control. It complements existing religious-domain data and supports evaluation across African languages and domains.
- MAFAND-MT introduces news-domain data for African languages, including six predominantly Francophone and five predominantly Anglophone languages.
- The corpus was created by crawling local newspapers, professionally translating 5k–8k sentences, and applying native-speaker quality control.Checks targeted misspellings, duplicated sentences, and alignment problems.
- Participatory curation assigned native speakers with societal knowledge to collection, translation, coordination, and quality-control roles.The process was designed to keep material appropriate and relevant for stakeholders of the resulting models.
- The corpus complements existing news resources for Hausa, Igbo, Swahili, Yorùbá, and isiZulu.
- Continued pre-training used monolingual text from 17 higher-resourced African languages plus Arabic, English, and French.
5 Models and Methods
The study adapts approximately 500M-parameter multilingual models and bilingual baselines for African news translation, including previously unseen languages. It compares continual pre-training, many-to-many translation, and several combinations of religious- and news-domain parallel data.
- Baseline Models: Pre-trained models include MT5, ByT5, mBART50, and M2M-100, while Transformer models provide bilingual baselines trained from scratch.MT5-base and ByT5-base are used alongside M2M-100 with 418M parameters.
- Transfer Learning Across Languages: The experiments compare zero-shot transfer, continual pre-training, and multilingual fine-tuning on parallel data for previously unseen languages.The section also evaluates models adapted to African languages through continual pre-training.
- Transfer Learning Across Languages: Continual pre-training adapts existing models with monolingual African-language corpora before fine-tuning on parallel translation data.The original training objective and vocabulary are retained during monolingual pre-training.
- Transfer Learning Across Languages: M2M-100 is fine-tuned into English- and French-centric models for multilingual translation across specified African-language groups.Languages absent from M2M-100 receive the language code of an included language outside the study.
- Domain Transfer: For limited news-domain data, the study compares joint REL+NEWS training, sequential REL→NEWS training, and additional NEWS fine-tuning after REL+NEWS.Each fine-tuning stage lasts three epochs, and translation quality is evaluated with BLEU and ChrF.
6 Results and Discussion
Fine-tuning multilingual pre-trained models on a few thousand high-quality translations adapts them to previously unseen African languages and improves transfer across domains. Domain-specific data and continued African-language pre-training further strengthen performance, although domain shift and evaluation limitations remain.
- 6.1 Adaptation to the Focus Languages: Zero-shot M2M-100 translation was very poor for most languages, while Swahili and Zulu benefited from greater pre-training exposure.Most languages scored below 5 BLEU, compared with over 13 BLEU for Zulu and over 20 BLEU for Swahili.
- 6.1 Adaptation to the Focus Languages: Fine-tuning multilingual pre-trained models on a few thousand sentences achieved strong performance, including for languages absent from pre-training, with M2M-100 performing best overall.M2M-100 was best overall, likely because it was pre-trained on translation; however, BLEU remained below 15 for many languages.
- 6.1 Adaptation to the Focus Languages: Continual pre-training improved AfriMT5 by 1.3 BLEU and 2.4 BLEU on average in the two translation directions, while AfriByT5 gains were 0.6 and 0.9 BLEU.Some languages not included in continual pre-training also benefited, possibly because of language similarity.
- 6.2 Adaptation to the News Domain: Fine-tuning on NEWS was more effective than adding larger religious-domain data, and REL→NEWS improved en/fr-xx performance by 1.1 BLEU.For xx-en/fr, REL+NEWS→NEWS was best, improving over NEWS and REL+NEWS by 2.0 and 1.5 BLEU.
- 6.2 Adaptation to the News Domain: Religious-domain models suffered severe NEWS degradation when religious training data was sparse, including drops of 95.5% BLEU for bbj and 93.5% for bam and luo.Larger religious corpora reduced, but did not eliminate, the domain-transfer loss.
- 6.3 Analysis of Domain Shift: Using 2.5k NEWS sentence pairs improved transfer across domains by up to 17 BLEU and also improved performance on Wikipedia and religious evaluation data.The authors report this pattern for both seen and previously unseen languages.
7 Conclusion
The paper introduces MAFAND-MT, a 16-language African news corpus, and finds that a few thousand high-quality sentences can adapt pre-trained models for low-resource translation.
- 2k sentences can suffice for fine-tuning, improving translation without large web-sourced text collections.The conclusion presents this as especially relevant for languages spoken by millions but poorly represented on the web.
- MAFAND-MT provides a corpus for studying low-resource translation systems in the African news domain.The corpus covers 16 African languages.
- The work is being expanded to Amharic, Chichewa, Kinyarwanda, Shona, and isiXhosa, alongside a larger Hausa corpus.
D Model Hyper-parameters and Reproducibility of Results
The experiments use standardized fine-tuning settings for pre-trained multilingual models, with additional pre-training procedures for AfriMT5, ByT5, and mBART50.
- Fine-tuning uses a 5e-5 learning rate, batch size 10, maximum sequence lengths of 200, beam size 10, and three epochs.NEWS-only models use 10 epochs, and experiments run on a single Nvidia V100 GPU.
- mBART50 decoding uses a supported closely related language when the target African language is absent from pre-training.Swahili is used as a convenient target language for unsupported African languages, while M2M-100 uses the correct language code when available.
- AfriMT5 and AfriByT5 begin from MT5 and ByT5 and receive one epoch of additional pre-training with 10,000 warm-up steps.mBART50 is additionally pre-trained for 50,000 steps without modifying its vocabulary.
E BLEU vs spBLEU
The paper compares BLEU and spBLEU across Wikipedia, religious, and news domains, finding higher spBLEU scores particularly for English or French into African languages.
- Tables 12 and 13 compare BLEU and spBLEU across Wikipedia, REL, and NEWS domains before and after NEWS fine-tuning.
- spBLEU gives higher scores than BLEU especially for en/fr-xx, suggesting it may better evaluate African languages.The authors state that further analysis and human evaluation are needed before establishing this generally.
- BLEU and spBLEU show little difference in the xx-en/fr translation direction.
F Qualitative Analysis
Qualitative examples show that M2M-100 produces more meaningful translations than a bilingual Transformer, while still making lexical, syntactic, and semantic errors.
- M2M-100 produces a more meaningful Fon-to-French translation than the bilingual Transformer in the qualitative example.The bilingual output is described as very poor and far from the Fon source, whereas M2M-100 preserves more meaning.
- M2M-100 preserves logical content and sentence style in French-to-Fon translation despite imperfect structure.The example retains the crucial negated final clause, though the prose notes structural and morphological challenges.
- The M2M-100 output contains misspelled names, incomplete expressions, and misplaced words that corrupt syntax and logic.
G Limitations and Risks
The study identifies limitations in translation quality, evaluation, language bias, domain coverage, language coverage, and potential harms from inaccurate or biased outputs.
- Low BLEU scores persist for bbj, mos, and zul, particularly when translating into these languages.
- BLEU and ChrF may miss semantic relatedness and can be inflated by word matches in incoherent phrases without deeper human evaluation.
- Language-specific differences in resources, morphology, standardization, pre-training inclusion, corpora, and cross-lingual relatedness still affect relative translation quality.
- Automatically estimated translation quality drops when the test domain is narrow, motivating evaluation across multiple domains.
- The study covers only 16 African languages, while thousands of other African languages remain unrepresented.
- The models may reproduce inaccurate or biased translations from pre-training data, and rapid fine-tuning could be misused for harmful purposes.
H.2 Additional experiments
Additional experiments examine domain transfer and Hausa corpus generalization, showing that adding contemporary news data substantially improves cross-domain performance in key directions.
- Domain shift: Figure 3 compares M2M-100 models trained on REL data and tested on NEWS versus REL across additional languages and translation directions.
- Generalization of Hausa news corpus: M2M-100 trained on the NEWS domain generalized poorly to REL and FLORES when based on the original Hausa Khamenei corpus.
- Evaluation dataset creation: Table 14 lists the added languages alongside their families, speaker counts, news sources, and NEWS and REL data splits.
- Generalization of Hausa news corpus: 4.0 →13.0 spBLEU on FLORES and 3.7 →8.8 on REL for EN-HAU followed addition of 2,767 contemporary news sentences.The gains came from adding Premium Times and Global Voices news data, although FLORES xx-en/fr performance dropped slightly.