Source-linked AI summary
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Hassan Muhammad, Salomon Kabongo, Salomey Osei, Sackey Freshia, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa, Mofe Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi, Laura Jane Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkabir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Espoir Murhabazi, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Emezue, Bonaventure Dossou, Blessing Sibanda, Blessing Itoro Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, Abdallah Bashir
TL;DR
NLP remains geographically and linguistically concentrated, and scaling MT to low-resourced languages is not adequately solved because low-resourcedness reflects more than data scarcity. This paper proposes participatory research involving necessary MT-development agents and demonstrates it through African-language MT, producing datasets, benchmarks, human evaluations, and scientific contributions from participants without formal training.
Problem
NLP has limited geographic and linguistic diversity, while low-resourcedness reflects scarce resources and broader societal constraints affecting MT development.
Method
The paper uses participatory research to involve necessary agents and interactions in sustainable MT development, tested through an African-language case study.
Results
The case study produced benchmarks and datasets for over 30 African target languages, with human evaluation enabled for many previously unevaluated languages.
Takeaways & Limitations
Participatory community design supported distributed growth, communication, knowledge sharing, model building, and human evaluation for low-resourced African MT.
Takeaways & Limitations
Existing low-resourced MT is constrained by limited content creation, technology access, and noisy or incorrect translation pairs caused by coding, orthography, or alignment issues.
Abstract
from arXiv · showhide
Research in NLP lacks geographic diversity, and the question of how NLP can be scaled to low-resourced languages has not yet been adequately solved. "Low-resourced"-ness is a complex problem going beyond data availability and reflects systemic problems in society. In this paper, we focus on the task of Machine Translation (MT), that plays a crucial role for information accessibility and communication worldwide. Despite immense improvements in MT over the past decade, MT is centered around a few high-resourced languages. As MT researchers cannot solve the problem of low-resourcedness alone, we propose participatory research as a means to involve all necessary agents required in the MT development process. We demonstrate the feasibility and scalability of participatory research with a case study on MT for African languages. Its implementation leads to a collection of novel translation datasets, MT benchmarks for over 30 languages, with human evaluations for a third of them, and enables participants without formal training to make a unique scientific contribution. Benchmarks, models, data, code, and evaluation results are released under https://github.com/masakhane-io/masakhane-mt.
1 Introduction
Low-resourcedness reflects both limited digital resources and broader societal and geographic inequities. The paper proposes participatory research for sustainable low-resourced MT and demonstrates it through an African-language case study.
- “Low-resourced” languages have few digital or computational resources, but the term also encompasses categories such as low density, less commonly taught, and endangered.The paper links language-resource scarcity to the societies and places where languages are spoken.
- NLP faces a diversity crisis involving limited geographic and linguistic coverage, prompting research groups and workshops focused on low-resourced languages.
- The study scopes its analysis to machine translation using parallel corpora.
- Participatory research involves necessary agents and interactions in sustainable MT development for low-resourced languages.
- The African-language case study produces novel translation datasets, benchmarks for over 30 target languages, and publications by participants without formal scientific training.
2 Background
Prior low-resourced MT work uses transfer, multilingual training, and additional monolingual data to improve translation, but benchmark coverage and evaluation remain constrained. The paper emphasizes that standard automatic scores may not reliably reflect translation quality without human evaluation.
- Cross-lingual Transfer: Cross-lingual transfer methods seek to transfer progress from high-resourced to low-resourced languages, but benchmark inclusion depends on available monolingual data and previously annotated resources.
- Multilingual Approaches: Multilingual MT trains models across languages simultaneously and often outperforms bilingual training, especially for low-resourced languages.
- Multilingual Approaches: English-centered multilingual parallel data creates a divergence between translation from English and translation into English.
- Multilingual Approaches: mBART combines monolingual and parallel data and improves translation quality for lower-resource Nepali, Sinhala, and Gujarati.These languages have more digital resources and a longer written-text history than the languages addressed by this paper.
- Targeted Resource Creation: Human evaluation remains important because the extent to which standard BLEU scores reflect translation quality is unclear.The cited work notes that human evaluation studies are missing.
- Targeted Resource Creation: Targeted low-resourced MT datasets expose challenges in tokenization, content selection, translation verification, and morphologically rich languages.The described study does not involve data curators or evaluators.
3 The Machine Translation Process
The MT process depends on interacting agents, resources, constraints, and stakeholder demand, but low-resourced settings often lack the societal, linguistic, and technical conditions needed for a sustainable cycle. The paper proposes participatory research that involves relevant agents, including participants from language communities, to address these limitations.
- The MT Process: Content creators, translators, and curators form dataset creation, while language technologists and evaluators form model creation; stakeholders create demand for both.Stakeholders also need access to technology and electricity to benefit from MT systems.
- The MT Process: Curators select dataset content and check language quality, language technologists build MT models, and evaluators measure performance and provide feedback.Their roles require language knowledge, data or translations, computational resources, metrics, and evaluation datasets.
- The MT Process: A high-resource English–French pair forms a virtuous cycle in which content supports MT, which drives more translation, tools, evaluation, and content.The cycle feeds back into improving MT systems.
- Limitations of Existing Approaches: Low-resourced MT is constrained by limited stakeholder demand, inadequate technology access, scarce digital content, and small or domain-specific translation resources.These constraints contribute to a chicken-or-egg problem in which more digital content could attract stakeholders and incentivize further creation.
- Limitations of Existing Approaches: Excluding local data curators can produce errors in encoding, orthography, and alignment, while high-resource techniques may fail under linguistic, infrastructural, compute, or data constraints.Evaluators may also rely on potentially unsuitable automatic metrics when human language expertise or crowdsourcing infrastructure is unavailable.
- Participatory Research Approach: The paper proposes participatory research involving agents from the languages’ countries or enabling knowledge transfer, and demonstrates the approach through African-language MT.The approach is intended to support faster and more effective iteration in the MT process.
4 Case Study: Masakhane
Masakhane addresses severe gaps in African-language NLP by using participatory research to connect distributed participants and produce MT resources and evaluations. The case study reports community growth, public research artifacts, and human-evaluation findings that expose limitations of standard benchmark metrics.
- Motivation: Africa’s 2144 living languages are poorly represented in NLP, with most lacking annotated data and even monolingual resources sparse.
- Motivation: Only five of 2695 affiliations at five major NLP conferences in 2018 came from African institutions, leaving researchers disconnected across the continent.
- Participatory approach: Masakhane treats participatory research as an empirical experiment that brings distributed participants together to train, contribute, and evaluate models.
- Research outcomes: The community grew to over 400 participants from at least 20 countries, while 86.5% reported that it helped them find mentors or collaborators.
- Research outcomes: Mentorship and knowledge exchange produced public datasets, benchmarks, and models, and enabled more than 10 participants to publish language-specific research.
- Human MT evaluation: Evaluation was difficult because Igbo required solutions for diacritics, dialect variation, terminology, and standardization.
- Human MT evaluation: Human evaluation revealed that JW300 scores can overestimate model quality, while training-data size appeared more reliable for predicting generalization.
- Human MT evaluation: Across languages and domains, evaluation performance varied substantially despite comparable training-data quantities and JW300 scores.
5 Conclusion
The paper concludes that participatory research can sustainably scale NLP research for low-resourced languages. The African MT community produced benchmarks, datasets, human evaluations, and broader collaboration while revealing strategies for distributed growth and knowledge exchange.
- Participatory research was proposed as a solution for sustainably scaling NLP research to low-resourced languages.
- The African MT implementation developed strategies for distributed growth, communication, knowledge sharing, and model building.
- The community published benchmarks and datasets for previously understudied languages and conducted human evaluation of model outputs.
- The volume and diversity of participants, languages, and outcomes included first human evaluations of MT systems for many featured languages.
A Demographics
A February 2020 voluntary survey found diverse educational and occupational backgrounds among participants, with undergraduate students forming the majority.
- A February 2020 voluntary survey found diverse educational backgrounds and occupations, with undergraduate students forming the majority.The community had grown by 30% by May 2020, so the figures represent a snapshot.
B Evaluation Data
Table 3 reports how many sentences were post-edited in the human evaluation study.
- Table 3 reports the number of sentences post-edited in the human evaluation study.
C Benchmark Scores
The paper reports benchmark evaluation details for JW300-based translation models alongside human-evaluation sentence counts. BLEU is computed with Sacrebleu without tokenization, and languages are categorized using Joshi et al. (2020).
- BLEU scores in Table 4 are computed with Sacrebleu using tokenizer ’none’ on JW300 test sets.JW300 data are tokenized with Polyglot, motivating the stated evaluation-tokenizer choice.
- Table 4 records benchmarks as of Nov 6, 2020 and categorizes target languages according to Joshi et al. (2020), as of 28 May 2020.