PaperlayerView exact revision on arXiv

Source-linked AI summary

No Language Left Behind: Scaling Human-Centered Machine Translation

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Jeff Wang

arXiv:2207.04672v3cs.CLcs.AI

TL;DR

Most machine-translation progress has focused on high-resource languages, leaving low-resource languages constrained by limited data and attention. NLLB combines community-informed research, data mining, multilingual modeling, and broad evaluation to build translation systems for more than 200 languages. Its models improve low-resource performance while the project also evaluates safety and documents important data and domain limitations.

  • Problem

    Most machine-translation improvements target high-resource languages, while low-resource languages remain underserved by limited data and research attention.

  • Method

    NLLB combines human-centered research with mined and curated data, multilingual Mixture-of-Experts modeling, regularization, curriculum training, and backtranslation.

  • Results

    NLLB creates over 1.1 billion new sentence pairs for 148 languages and reports strong gains on low-resource directions while maintaining high-resource performance.

  • Takeaways & Limitations

    The project evaluates translation across thousands of directions with calibrated human assessment and toxicity analysis, supporting broader assessment of quality and safety.

  • Takeaways & Limitations

    Mined datasets are limited by scarce monolingual data, filtering effects, uneven language impact, duplicated content, and likely machine-translated web material.

Abstract

from arXiv · show

Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today. However, such efforts have coalesced around a small subset of languages, leaving behind the vast majority of mostly low-resource languages. What does it take to break the 200 language barrier while ensuring safe, high quality results, all while keeping ethical considerations in mind? In No Language Left Behind, we took on this challenge by first contextualizing the need for low-resource language translation support through exploratory interviews with native speakers. Then, we created datasets and models aimed at narrowing the performance gap between low and high-resource languages. More specifically, we developed a conditional compute model based on Sparsely Gated Mixture of Experts that is trained on data obtained with novel and effective data mining techniques tailored for low-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Critically, we evaluated the performance of over 40,000 different translation directions using a human-translated benchmark, Flores-200, and combined human evaluation with a novel toxicity benchmark covering all languages in Flores-200 to assess translation safety. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art, laying important groundwork towards realizing a universal translation system. Finally, we open source all contributions described in this work, accessible at https://github.com/facebookresearch/fairseq/tree/nllb.

1. Introduction

Machine translation has advanced unevenly, benefiting high-resource languages while most low-resource languages remain underserved by limited data and research attention. NLLB addresses this gap with human-centered research, large-scale data creation, multilingual models, comprehensive evaluation, and open-source releases.

  • High-resource languages have received most machine-translation improvements, while the majority of global languages remain low-resource and underserved.
  • Low-resource translation is constrained by scarce, expensive training data and the difficulty of scaling data-hungry techniques.
  • NLLB frames its effort around understanding native speakers, creating training data, building translation models, and evaluating every target language.
  • The project combines human-translated and mined data with multilingual Mixture-of-Experts and dense Transformer models covering 202 languages.
  • The work open-sources its benchmarks, data, scripts, and models, and deploys translation support for Wikipedia editors in underserved languages.
  • Its releases include Flores-200 in 204 languages, seed datasets, Toxicity-200, language-identification and bitext-mining tools, and reproducibility scripts.

2. Human-Centered Low-Resource Language Translation

NLLB grounds low-resource translation in community perspectives and responsible-AI principles, while addressing concerns about quality, safety, representation, and cultural preservation. Interviews informed an interdisciplinary research process and practical attention to community needs.

  • The project uses Value Sensitive Design to place community interests, user interaction, and ethical and social considerations at the center of its research.
  • Researchers interviewed 44 native speakers across 36 low-resource languages, defining low-resource languages as having fewer than 1 million public example translations.
  • Participants linked language support to cultural preservation, including creating and accessing cultural knowledge through Wikipedia.
  • Concerns included excluding language variants or scripts and privileging better-resourced forms as defaults.
  • Interviewees emphasized that unreliable, low-quality, or toxic translations can discourage use even when a language is technically included.
  • The research adopts four guiding principles informed by interviews and responsible-AI literature, including interdisciplinary collaboration across humanities, social sciences, and technical fields.

3. Languages

NLLB expanded language coverage from Flores-101 toward 200+ languages by combining community-informed selection with systematic resource and language documentation. The final set emphasized potential community impact, written standardization, and support for multiple scripts and variants.

  • Language selection: Coverage expanded from 101 to 200+ languages after evaluating over 250 candidates and selecting around 210 for final expansion.
  • Language selection: The preliminary list began with languages having a Wikipedia presence and incorporated native-speaker regional input, existing dataset coverage, and gaps in evaluation resources.
  • Community considerations: Language inclusion was assessed by potential impact on low-resource communities, while languages with extremely few native speakers or predominantly oral use were excluded.
  • Language documentation: The project documented language codes, scripts, families, subgroupings, web support, resource levels, and regional or variety specifications for the supported languages.
  • Language documentation: A language was classified as low-resource when it had fewer than 1M publicly available, de-duplicated bitext samples with any language in the 200-language set; 150 languages met this criterion.

4. Creating Professionally Translated Datasets: FLORES-200 and NLLB-Seed

The authors created professionally translated resources to address low-resource data scarcity, including the many-to-many FLORES-200 benchmark and seed datasets for model training and evaluation. A staged workflow combined professional translation, review, automated checks, and post-editing, while quality and dataset limitations remained important considerations.

  • Dataset contributions: FLORES-200 doubles Flores-101’s language coverage and, with professionally translated seed data in 39 low-resource languages, supports model creation and evaluation.
  • Translation workflow: The translation workflow aligned translators and reviewers, translated the sentence set, performed automated checks, conducted independent quality review, and used post-editing when needed.
  • 4.1 FLORES-200: FLORES-200 contains 3001 sentences from English-language Wikimedia projects, professionally translated into 200+ languages as a many-to-many benchmark.
  • Translation workflow: Professional translators were required to be native speakers, fluent in English, and experienced in the relevant language pair, with periodic translation testing.
  • Quality and limitations: Human translation quality was affected by mistranslations, unnatural phrasing, borrowings, literal translations, and spelling inconsistencies linked to limited standardization.
  • Quality and limitations: The datasets are English-centric and may underrepresent culturally diverse content, while mined web data may include content that was already machine translated.

5. Automatically Creating Translation Training Data for Hundreds of Languages

The section presents a pipeline for creating clean monolingual data and mined bitext for hundreds of languages, combining language identification, filtering, and LASER3-based alignment. Teacher-student encoders and language-specific training improve mining and translation quality, although scarce web data still limits some languages.

  • 5.1 Language Identification: Web noise, parsing artifacts, and the mismatch between clean training data and noisy inputs motivate human auditing of language identification.The challenge includes language mixing, creative scripts, leetspeak, and incomplete or grammatically incorrect strings.
  • 5.1 Language Identification: The language identification system covers 200 languages and outperforms LangId, LangDetect, and CLD3, while also improving False Positive Rate.The model supports all 200 Flores-200 languages, compared with 107 for CLD3.
  • 5.2 Monolingual Data: The pipeline filters monolingual web data before mining, discarding 2.8 billion sentences and retaining 43.7 billion sentences for bitext extraction.The filtering uses language-identification, script, language-model, and cleaning criteria, with aggressive filtering for high-resource languages.
  • 5.3 Student-Teacher Mining Approach: LASER3 uses teacher-student training to adapt a multilingual sentence encoder to low-resource languages without competing for capacity with high-resource languages.The approach preserves compatibility of multilingual embedding spaces for subsequent mining and avoids retraining the full model from scratch.
  • 5.3 Language-Specific Encoder Training: Teacher-student and language-specific encoders substantially reduce xsim error and enable strong BLEU scores for several low-resource languages.Average xsim error falls from 61 to 0.91; examples include Javanese improving from 11.1 to 31.2 BLEU and Sundanese from 1.5 to 28.5.
  • 5.3.5 Limitations of Large-Scale Mining: Mining remains constrained for languages with little monolingual web data, with compounded filtering and alignment challenges affecting languages unevenly.Kabyle reaches 6.2 BLEU, while insufficient monolingual data makes high-quality Tamashek bitext impossible to mine.

6. Modeling

The modeling section scales multilingual translation with conditional computation and regularization to address capacity interference and overfitting across hundreds of language directions. MoE variants, expert-output masking, conditional routing, curriculum learning, and expert specialization analyses identify strategies that improve low-resource performance while preserving broader quality.

  • Conditional computation: Conditional compute models provide capacity for hundreds of languages while reducing interference between unrelated translation directions.The approach uses sparsely activated expert subnetworks rather than activating all model parameters for every input.
  • MoE scaling: 1+ chrF++ improvements occur across most subsets for MoE-64 versus dense 615M, excluding very low-resource and non-English pairs.The computationally similar MoE-64 model improves all reported subsets except v.low and xx-yy.
  • Regularization: +1.5 to +1.7 chrF++ across all subsets results when MoE-64 with pdrop=0.1 is compared with dense 615M with dropout.Increasing dropout from 0.0 to 0.1 also changes the relative very-low-resource result from -0.1 chrF++ to +0.9 chrF++.
  • Conditional routing: Conditional MoE Routing learns whether tokens use a shared dense FFN or specialized MoE experts, improving CMR results over vanilla MoE with overall dropout.CMR achieves +0.8 chrF++ into English and +0.7 chrF++ on non-English pairs, with larger gains for low and very low-resource languages.
  • Regularization: MoE Expert Output Masking improves regularization and reduces overfitting on low-resource tasks compared with overall dropout.EOM gains +0.4 chrF++ into English and +0.6 chrF++ on non-English pairs over vanilla MoE with overall dropout.
  • Curriculum learning: Curriculum learning improves full-dataset training when combined with EOM, while EOM alone already reduces overfitting on the ablation dataset.On the ablation dataset, adding curriculum to EOM hurts performance, whereas the full dataset and larger model benefit, especially on low-resource pairs.

7. Evaluation

NLLB evaluates translation quality with calibrated human judgments alongside automatic metrics, and assesses safety through toxicity detection across languages. The evaluation examines metric correlations, translation-direction differences, quality distributions, and limitations in toxicity measurement and mitigation.

  • Human Evaluation Methodology: 0.710, 0.687, and 0.694 are the Spearman R correlations of aggregated XSTS with spBLEU, corpus chrF++, and sentence-level chrF++, respectively.Calibrated human scores correlate more strongly with automated metrics than uncalibrated scores.
  • Evaluation Results: Translation directions show three quality patterns: consistently strong output, variable quality with poorly rated sentences, and many poor or incoherent translations.The distributions distinguish high average scores from directions containing substantial low-quality or incomprehensible output.
  • Evaluation Results: Into-English translation generally receives higher human-evaluation performance than out-of-English translation, consistent with chrF++ and spBLEU.The comparison is made across translation directions in Figure 26.
  • Toxicity Evaluation: 66.8% F1 in English and 61.6% F1 in French were achieved by the toxicity detector, which performed better than chance on sampled languages.Detector performance decreased for detecting multiple toxic items, and a significant difference remained between high- and low-resource languages.
  • Toxicity Evaluation: Misaligned mined bitext can encourage mistranslations with added toxicity when toxic content appears on only one side of the training pair.The sampled corpora contained substantial misalignment, often through omission or detoxification on one side.
  • Human Evaluation Methodology: XSTS and evaluator calibration provide more comparable human-quality scores across evaluators and language pairs.Calibration adjusts raw scores using a shared English calibration set and improves correlation with automated metrics relative to uncalibrated scores.

8. Bringing it All Together

NLLB combines transparent, human-centered data work with mining, backtranslation, filtering, and multilingual model scaling to improve translation across more than 200 languages. Across Flores-101 and Flores-200 evaluations, the resulting system substantially improves over prior state of the art while retaining gains for low-resource directions.

  • Data and benchmark: +18.6 chrF++ overall improves very-low-resource-to-English directions, including +12.5 chrF++ from mined data and +6.1 chrF++ from additional backtranslated data.Very low-resource languages are defined as having fewer than 100K unique sentence pairs across available public bitext corpora.
  • Data and benchmark: +8.9 chrF++ improves non-English-centric very-low-resource pairs, while out-of-English directions improve by +6.6 chrF++ overall.For non-English-centric pairs, mined data contributes +7.5 chrF++ and backtranslated data +1.4 chrF++; for out-of-English directions, the contributions are +4.7 and +1.9 chrF++.
  • Data and benchmark: NLLB combines mined, human-translated, backtranslated, and filtered data into a final dataset covering 202 languages.The effort integrates interdisciplinary work and shares its roadblocks, successes, and training resources transparently.
  • Model scaling and training: MoE layers every 4 Transformer blocks are the best-performing configuration and are 28% more efficient than placing them every 2 blocks.The improvement is especially pronounced in very-low-resource settings.
  • Model scaling and training: Curriculum learning outperforms no curriculum on the massively multilingual dataset, where thousands of low- and very-low-resource directions increase overfitting risk.The authors attribute the difference from earlier experiments to the dataset’s fourfold increase in languages and many additional directions.
  • Evaluation: +7.3 spBLEU on average, a 44% improvement, lets NLLB-200 outperform the nearest state of the art on Flores-101 despite covering 200+ languages.On a smaller 87-language subset, it also improves by +6.9 spBLEU on average.

9. No Language Left Behind: Social Impact & Concluding Thoughts

NLLB could expand information access and linguistic participation, but its social benefits depend on addressing risks, inequities, and broader policy needs. The authors therefore frame NLLB as one component of a multifaceted effort rather than a complete solution.

  • Contributions: Flores-200 and NLLB-Seed address major evaluation and training-data gaps for building translation systems covering 200+ languages.The datasets are presented as central contributions for improving low-resource translation research.
  • Potential Benefits: NLLB could improve web access, education, cultural exchange, and knowledge sharing for low-resource language communities.The authors also describe benefits for communities already served by translation through improved quality and easier information access.
  • Risks: Greater digital participation may increase exposure to surveillance, censorship, misinformation, scams, and hate speech.The authors suggest pairing translation tools with harmful-content detection, online literacy, and structural investments.
  • Risks: NLLB may worsen local digital inequities because benefits depend on technological know-how, socioeconomic conditions, rurality, and uneven infrastructure.The distribution of access and ability to use the technology constrains its impact.
  • Open Research: Open-sourcing NLLB’s datasets and models supports transparency, reduces duplicated effort, and enables broader scientific collaboration.The project also aims to help researchers adapt the models and identify remaining gaps.
  • Conclusion: NLLB remains only one technological intervention, so policy action on education, internet access, and digital literacy is also required.The conclusion emphasizes collaboration with stakeholders to make the web more accessible across languages.

10. Contributions

The Contributions section attributes NLLB’s work across data, modeling, evaluation, ethics, engineering, and project leadership. It emphasizes that the effort depended on coordinated contributions spanning multiple technical and disciplinary areas.

  • Data: The data team built monolingual cleaning, language-identification, mining, deduplication, and infrastructure pipelines for low-resource language data.Contributors also worked on LASER3, mined bitext, data partnerships, and open-sourcing mined data.
  • Modeling: The modeling team led research and experimentation on Mixture-of-Experts architectures, backtranslation, self-supervised learning, and model-quality analysis.These contributions supported the project’s effort to scale translation across 200 languages.
  • Evaluation and Ethics: Evaluation and ethics contributors handled toxicity analysis, interpretability, data sheets, human annotations, interview studies, and human-study design.Other work covered transliteration, dialectal variation, language information, output quality, and toxicity detection.
  • Leadership: Project and engineering leaders provided overall research direction, management, open-source strategy, and guidance for data and evaluation research.The section also lists product and program-management responsibilities.

A.1 Ethical Considerations around Language Standardization

NLLB’s language coverage includes languages with centralized or distributed standardization, making translation development partly a question of linguistic representation and community power. Annotation choices can favor some variants while transfer and adaptation may extend support to others.

  • Responsibility: Because standardization is tied to power and identity, language technology can influence how languages are used and evolve.The authors therefore examine responsibility through contrasting centralized and distributed cases.
  • Standardization Types: Language standardization may be centralized through governmental, academic, or community institutions, or distributed across competing bodies and weakly standardized variants.These categories reflect differing organizational resources and levels of recognized authority.
  • Risks of Annotation: Annotating datasets for distributedly standardized languages requires guidelines that may favor one variant and make other groups feel excluded.The nearest boundary is that out-of-the-box support may cover only selected variants.
  • Adaptation: Related languages can benefit from transfer learning, while transliteration and spelling-standard mappings can extend support across variants with limited additional annotation.The passage presents these as ways competing variants may benefit from appearing together in the dataset.

Appendix B. Evaluation

The toxicity benchmarks use human translation of English prompts, with prompt design varying across language groups. The authors identify persistent ambiguity in judging slurs and nontoxic language, especially across cultural contexts.

  • Benchmark Construction: Toxicity lists for non-English languages were created through human translation of an English prompt, with German and Spanish pivots for Luxembourgish and Asturian.The prompt was iteratively modified to reduce errors from ambiguity and misunderstanding.
  • Benchmark Construction: Earlier FLORES 101 languages used prompts without definitions, part-of-speech information, or register information, while allowing cultural and spelling expansions.These lists could include inflections, leetspeak, and nonstandard variants.
  • Benchmark Construction: For newly added languages, prompts included definitions, part-of-speech and register information, and encouraged translators to identify missing equivalents and culturally specific items.Suggestions were limited to around forty items without specific restrictions.
  • Limitations: Translators still faced similar difficulties deciding whether some items were slurs or nontoxic language despite richer prompts and complementary training.The difficulty was especially salient for some racial slurs and reclaimed terms.

Appendix C. Data

This appendix evaluates language identification using Flores-200 annotations and reports results across all supported languages, including a human-annotated challenge comparison with CLD3.

  • Flores-200 devtest provides the annotations used to assess language identification precision, recall, and F1 across all languages.The results are reported in Table 49.
  • Table 50 compares CLD3 with the proposed language identification model on a challenge set built from human annotations.False positive rates are reported alongside F1 scores, and unsupported CLD3 languages are left blank.
  • False positive rates are included because F1 can mislead when deployment data have different class balances from the development set.This caveat is specifically relevant when applying language identification to web data.

D.1 Ablation Dataset

The ablation appendix defines the language-pair dataset and examines how backtranslation sources and quality affect multilingual model scaling.

  • Ablation dataset: The ablation dataset contains exactly 110 language pairs used for experiments in Section 6.Table 51 lists the complete set of language pairs.
  • Backtranslation sources: Combining SMT and MMT backtranslations is beneficial because the two models provide different, complementary sources of noise.Token-frequency plots visualize differences between their generated translations for two directions.
  • Training data: The training-data summary identifies major datasets used for NLLB-200 and notes that reported direction counts exclude reverse directions.The data were largely downloaded through OPUS and mtdata.
  • Backtranslation quality: The study varies backtranslation-generating model size and final model size across eight African languages and 60 directions involving French and English.Generating models are 219M or 1.4B parameters, while final models range from 219M to 1.4B parameters.
  • Backtranslation quality: Performance quickly plateaus when training uses backtranslations generated with small models.The experiment investigates this effect across multiple final model sizes.

E.1.3 Training Directions and Curriculum Buckets.

This appendix documents training-direction resources, curriculum buckets, benchmark comparisons, safety evaluation, model documentation, and intended-use boundaries for NLLB-200.

  • Training directions and curriculum: Training directions are organized into Primary, Mined, Primary+Mined, and Primary+Mined+MmtBT+SmtBT buckets.The appendix also lists files for a four-phase curriculum and its language-pair configurations.
  • Benchmark comparisons: NLLB-200 outperforms MMTAfrica on most non-English-centric Flores-101 translation directions.The comparison reports spBLEU/chrF++ and identifies NLLB-200 as the best system on most directions.
  • Benchmark comparisons: NLLB-200 performs better than Google Translate on xx-eng_Latn directions and on the overall average, with larger gains on low- and very-low-resource pairs.Google Translate performs better on eng_Latn-xx directions, particularly several high-resource directions.
  • Safety evaluation: Toxicity detections are dominated by benign-term over-detection, although some detections correspond to genuinely hallucinated toxicity.Out-of-English results are more inconsistent across languages than into-English results.
  • Intended use and limitations: NLLB-200 is intended for research and single-sentence translation among 200 languages, not production deployment, domain-specific or document translation, or certified translations.Inputs longer than 512 tokens may experience quality degradation.
  • Intended use and limitations: The model has been tested on Wikimedia with limited investigation of other supported domains and may not capture all language variations.Users are advised to make appropriate assessments.

Appendix G. Data Card for NLLB-Seed Data

NLLB-Seed is a human-translated dataset designed to provide clean multilingual data and improve translation quality and language identification for extremely low-resource languages.

  • Dataset construction: NLLB-Seed contains English-to-43-language bitext covering 6,193 sentences sampled from Wikimedia topics.The source topics come from a collection of 10,000 Wikidata IDs spanning knowledge and human activity.
  • Curation rationale: Translation procedures were aligned using Flores-200 conventions for script, dialect, spelling, and translation approach.Most languages were translated directly from English, with specified exceptions for Ligurian and some Arabic scripts.
  • Social impact: The dataset was built to improve translation quality and language identification for extremely low-resource languages.Its stated social-impact goal is improving the quality of these languages in machine translation systems.
  • Dataset limitations: Biases in the NLLB-Seed dataset have not been studied.
  • Dataset curation: Translators were required to be native speakers, highly fluent in English, and experienced in translation, with additional source-language requirements for non-English translators.The curation process included vetting by translation-vendor partners.

Appendix H. Data Card for NLLB Multi-Domain Data

The NLLB Multi-Domain dataset contains human-translated English-to-six-language bitext spanning four domains, designed to improve and assess translation across domain variation. Translations follow Flores-200 linguistic alignments and undergo quality assessment, while dataset biases have not been studied.

  • 11,810 human-translated sentences cover news, formal speech, informal speech, and medical domains across English and six other languages.The dataset supports evaluating general translation models and fine-tuning on new domains.
  • Translations were produced directly from English and checked against Flores-200 linguistic alignments through linguistic quality assessment and basic sanity checks.
  • 2,810 news sentences, 3,000 informal speech utterances, and 3,000 health sentences form the main domain-specific sources.The news data comes from WMT21 English-German development data; informal speech comes from a multi-session chat dataset; health data comes from World Health Organization and TAUS materials.
  • The dataset includes no extra annotations, and its translators were vetted as native target-language speakers with advanced English fluency and translation experience.
  • Dataset biases have not been studied.

Appendix I. Data Card for Mined Bitext Metadata

The mined-bitext release provides metadata and reconstruction scripts for large-scale language-pair data mined from public web sources. Because the mining process is noisy and may preserve web biases or sensitive content, the authors recommend using it for training rather than evaluation.

  • 1,613 language-pair datasets comprise mined bitext for 148 English-centric and 1,465 non-English-centric pairs.The release includes metadata and a script for recreating the full bitext from specified CommonCrawl and ParaCrawl files.
  • Metadata files record source locations, hashes, quality scores, directions, languages, and line numbers for each bitext direction.Common Crawl and other corpus sources use different metadata fields, while paragraph and sentence digests use xxh3_64_intdigest.
  • The authors recommend using the mined data for training and Flores-200 or other datasets for evaluation because of the process's noisy nature.
  • LASER3 encoders identified parallel sentences, alongside language-identification, emoji, and selected language-model filtering.
  • The data was not human annotated or curated, and publicly sourced text may contain personally identifiable, sensitive, or toxic content.
  • Biases have not been specifically studied, and web-source, language-identification, and filtering processes may introduce additional biases.
Loading 2207.04672v3…