Building Machine Translation Systems for the Next Thousand Languages

1 AN OVERVIEW

• The paper builds monolingual web-text corpora for 1500+ languages and audits 72 corpora, finding 70%–100% in-language data with a median score of 80%.Its mining approach scales LangID and applies document-consistency, wordlist, TF-IIF, and other filtering techniques.Source: §1 · p. 4
• The MT approach combines parallel data for about 100 higher-resource languages with monolingual data for 1000+ languages, plus self-supervised learning, data augmentation, and high-capacity multilingual models.The paper describes zero-resource translation as a technical setting in which the model sees no direct parallel text for the long-tail languages.Source: §1 · pp. 4–5
• The paper examines BLEU’s limitations, advocates CHRF for long-tail evaluation, proposes round-trip evaluation where references are unavailable, and reports human evaluations on 28 languages.The human evaluations support the conclusion that the described recipe can produce functioning MT systems.Source: §1 · p. 5
• Qualitative analysis finds confusion between distributionally similar words, weaker translation of infrequent tokens in lower-resource settings, and failures on short inputs; distilled models can magnify training-data biases or noise.Examples include translating “tiger” as a “miniature crocodile.”Source: §1 · p. 5
• Additional experiments report better performance for direct translation between similar languages than via English pivot, zero-shot transliteration, and a punctuation-based “period trick” that improves translation quality.The models are also reported as robust to nonstandard Unicode glyphs for some, but not all, languages.Source: §1 · p. 5
• 24 of the 30 languages considered for launch met the bar for Google Translate, after individual evaluation and consultation with affected communities.The selected languages include regions under-represented in technology, including the Americas, Africa, and South Asia.Source: §1 · p. 5

2 BUILDING A 1000-LANGUAGE WEB TEXT DATASET

• Web mining low-resource languages is difficult because LangID suffers from domain mismatch, n-gram pathologies, and class imbalance, while existing public multilingual corpora can be unusably noisy.The internet’s scale also makes computationally intensive filtering and storage of unfiltered data challenging.Source: §2 · p. 6
• The approach crawls monolingual text for 1500+ languages, prioritizing high-precision, in-language data through multiple filtering steps.It extends and improves on methods presented in Caswell et al. (2020).Source: §2 · p. 6

2.1.1 LANGID MODELING

• The work uses efficient CLD3 character-ngram models and semi-supervised LangID models; the latter use a Transformer Big architecture and are described as more accurate and robust to web noise, but slower.Three models serve training-data quality measurement, web crawling and clustering, and later filtering.Source: §2.1.1 · p. 6

2.1.2 PARING DOWN THE SET OF LANGUAGES

• After evaluating 1,745 language varieties against precision, confusion, crawl overtriggering, and training-data criteria, the authors removed 116 languages, primarily for poor-quality training data.They also omitted the 83 highest-resource languages from this crawl because of storage limits and existing monolingual data.Source: §2.1.2 · p. 7

2.1.3 FALSE NEGATIVE RATE CLUSTERING

• Clustering by LangID false-negative rate aims to preserve recall among related languages; although the median dataset-size ratio was 1.006, 57 languages gained more than 20× data.Large gains included related varieties such as Hindustani and Arabic varieties, and Oromo and Eastern Oromo.Source: §2.1.3 · p. 7

2.1.4 DOCUMENT CONSISTENCY FILTERING AND FIRST-PASS LANGID

• Document-consistency filtering keeps sentences whose cluster prediction matches the document’s most frequent cluster prediction.In the initial 1,745-language crawl, 83% of sentences had a consistency score under 10%; the authors suggest a lower threshold for retaining multilingual-page content as future work.Source: §2.1.4 · pp. 7–8

2.1.5 HIGH-RECALL WORDLIST-FILTERING

• Percent-threshold wordlist filtering discards sentences with less than 20% in-language words for any language in the cluster, using the 800 most frequent training-data words.Source: §2.1.5 · p. 8

2.1.6 FILTERING AND DECLUSTERING WITH SEMI-SUPERVISED LANGID

• After clustered first-pass filtering, SSLID classifies sentences; sentences predicted as any language outside the cluster are filtered, and CLD3 predictions are ignored.Source: §2.1.6 · p. 8

2.1.7 TF-IIF FILTERING

• TF-IIF filtering addresses severe pollution missed by earlier filters, including a Tok Pisin corpus that was over 99.7% Standard English; it retains sentences with at least 20% of tokens in TF-IIF wordlists.The authors omit the IDF term as it becomes less helpful as the number of languages scales, and use κ = 80000 and τ = 1000.Source: §2.1.7 · pp. 8–9
• Increasing κ steadily reduces distractibility with little recall loss, while increasing wordlist size τ recovers recall but raises distractibility.The authors define Relative Recall Rate to decide when to filter, weighting recall loss more heavily than dataset-size reduction because evaluation recall may overestimate recall on natural web text.Source: §2.1.7 · pp. 9–10
• With ρ = 2, the heuristic flagged 895 of 1503 languages; applying additional thresholds led to TF-IIF filtering for 210 corpora.Filtering required at least 20% of crawl data to be removed and at least 80% filtering recall.Source: §2.1.7 · p. 10

2.1.8 TOKEN DISTRIBUTION ANOMALY DETECTION AND NEGATIVE TOKEN FILTERS

• The approach targets anomalous data because some unwanted content is technically in-language and cannot be removed by TF-IIF or straightforward LangID.Examples included casino-heavy Darija data and unit-conversion templates in Arabic-script Indo-Aryan languages.Source: §2.1.8 · pp. 10–11
• The authors compared top-token distributions in crawled data with LangID training data, then combined two scores with a harmonic mean into the Harmonic Token Anomalousness Score.Jenson-Shannon divergence and Pearson R were considered but found ineffective.Source: §2.1.8 · p. 11
• Scores below 0.70 flagged questionable quality, while scores above 0.97 often indicated that the crawl had recovered training data, often religious material.The measure is approximate and should be used judiciously.Source: §2.1.8 · p. 11
• Manual review of 179 suspicious datasets led to filters for 62 languages; these removed a median 21% of data in those languages.Many flagged cases had no clear filtering solution and were left as-is.Source: §2.1.8 · p. 11

2.1.9 DEDUPLICATION

• Deduplication reduced median dataset size by 1.8x and average dataset size by 1.4x.Source: §2.1.9 · p. 11

2.2.1 MONOLINGUAL DATA

• The resulting corpus covers 1,503 low-resource languages, and experiments use 1,057 with more than 25,000 monolingual sentences before deduplication.Dataset sizes range from one sentence for Mape to 83 million for Sabah Malay; training data also includes 83 high-resource languages.Source: §2.2.1 · p. 11

2.2.2 MONOLINGUAL DATA QUALITY AUDIT

• The 72-language audit found a median data score of 80%, using weighted labels for correct, low-quality, ambiguous-dialect, and wrong-dialect samples.Wrong Language averaged 16% and Low Quality/Boilerplate averaged 10%; close dialects of major languages often had the poorest quality.Source: §2.2.2 · pp. 11–12

2.2.3 PARALLEL DATA

• The authors also used a web-crawled parallel corpus of approximately 25 billion sentence pairs spanning 112 languages to and from English.Source: §2.2.3 · p. 12

3 BUILDING MACHINE TRANSLATION MODELS FOR LONG-TAIL LANGUAGES

• The training recipe combines in-language monolingual learning, multilingual supervised translation, synthetic-data augmentation, and scaling to address a median of only about 43K monolingual sentences per language.Most evaluated languages use no in-language supervised data; Sorani Kurdish is the exception.Source: §3 · p. 13, §3.1 · p. 13

3.2 EXPERIMENTS COMPARING EFFECT OF MODEL CAPACITY

• Increasing model capacity from 1.5B to 6B improved xx→en by an average 2.7 CHRF but reduced en→xx by an average -1.0 CHRF.The en→xx regression was associated with models inferring the task from the input language and copying instead of translating.Source: §3.2 · p. 14

3.3 CONTINUAL LEARNING, EXTENDING MODELS FROM 200 TO 1000 LANGUAGES

• Expanding monolingual training from 206 to 1000 languages improved xx→en by +2.5 and en→xx by +5.3 average CHRF, with less benefit for languages lacking similar training languages.The authors suggest transfer between similar languages outweighed interference in this resource-constrained setting.Source: §3.3 · pp. 14–15, Table 4 · p. 16

3.4 EFFECTS OF LARGE SCALE DATA AUGMENTATION

• Large-scale augmentation substantially improved quality across the 30 fine-tuned languages; except for the Native American subset, most exceeded 40 CHRF xx→en and 35 CHRF in the reverse direction.The procedure used up to 10 million sentences per language and repeated offline augmentation twice.Source: §3.4 · p. 17, Table 5 · p. 17

3.5 EFFECT OF FILTERING SYNTHETIC DATA

• Round-trip translation and LangID filtering usually changed performance by no more than 0.5 CHRF, but improved Nigerian Pidgin and Kalaallisut outputs affected by training-data pollution.The identified contamination involved AAVE in Nigerian Pidgin data and Danish in Kalaallisut data, which could be magnified during self-training.Source: §3.5 · pp. 17–18, Table 6 · p. 18

3.6.1 DATA GENERATION FOR DISTILLATION

• For distillation, the teacher generates synthetic forward- and back-translated data to train smaller multilingual students; the shallow and deep encoder candidates have 330M and 850M parameters.Synthetic data is round-trip and LangID filtered, with additional manual regex filters for some language-specific noise.Source: §3.6.1 · pp. 18–19, §3.6.2 · p. 19

3.6.2 DISTILLATION APPROACH AND HYPER-PARAMETERS

• Most distilled models were similarly insensitive to tested hyperparameters, while the shallow encoder underperformed and deeper models often erased shallow-model trends.On shallow models, increasing xx→en back-translated data from 1M to 2M sentences raised CHRF by about +0.6, with minimal gains beyond 2M; en→xx showed no significant differences from 1M to 8M forward-translated sentences.Source: §3.6.2 · p. 19

3.6.3 COMPARISON AGAINST TEACHER MODEL

• +1.1 CHRF en→xx and +0.2 average CHRF xx→en separate the best deep-encoder students from their teacher, though the authors attribute much of the difference to English-original evaluation data.Translationese can lead reference-based CHRF to overestimate en→xx and underestimate xx→en performance.Source: §3.6.3 · pp. 19–20, Table 7 · p. 20

4.1 EVALUATION SETS; 4.2 EVALUATION METRICS

• CHRF is expected to correlate better with quality than BLEU for highly inflecting languages, but the metrics are difficult to compare and respond to different artifacts.The evaluation uses small, 1,200-sentence sets across 38 diverse languages; its approximate scaled-CHRF comparison highlights differing metric pictures, not directly comparable scores.Source: §4.2 · pp. 21–22, §4.1 · p. 21

4.3 RTT LANGID CHRF

• RTTLANGIDCHRF correlates moderately with CHRF across 30 languages, but its relationship varies by direction and improves for distilled models only in xx→en.Across the 630 languages with valid scores, scores generally rise with monolingual data size; the authors advise skepticism because high scores may not indicate high quality.Source: §4.3 · p. 23

4.4 HUMAN EVALUATIONS

• Human judgments show that automatic metrics can overestimate translation quality for related dialects, including outputs that native speakers considered unusable.The authors attribute these cases to data pollution, with outputs sometimes appearing in a corrupted or more common related variety.Source: §4.4 · pp. 23–24

4.5 MISTAKES ON DISTRIBUTIONALLY SIMILAR WORDS

• Zero-resource models often substitute distributionally similar words, including common nouns, and around 25% of languages translated “English Language” as their own language’s name.The authors also observed a gradient in color-term translation: red was often translated well, yellow less well, and orange poorly.Source: §4.5 · pp. 24–25

4.6 PERFORMANCE ON TOKENS BY TOKEN FREQUENCY

• Token hit rates can expose weaknesses hidden by CHRF: Fulfulde’s CHRF of 40.7 accompanies a drop from 41% on bin 2 to 26% on bin 4.The analysis bins English tokens by frequency and measures how many reference tokens from each bin appear in model outputs.Source: §4.6 · p. 26, Table 11 · p. 27

4.7 ERRORS ON SHORT INPUTS

• Single-word inputs often yield verbose translations, alternate outputs, appended frequent tokens, duplicates, or copied inputs, especially for lower-resource languages.Without source-language information, xx→en must infer language and translate simultaneously; the authors suggest source-language information as a possible remedy.Source: §4.7 · p. 27, Table 13 · p. 27

4.8 MAGNIFICATION OF ERROR MODES IN STUDENT MODELS

• Distilled outputs showed amplified errors, including wrong-language translations; specialized filters and retraining reduced these problems, which were more prevalent in synthetic data and noisier-source translations.The authors hypothesize that error magnification may reflect self-training feedback or domain differences between training and distillation data.Source: §4.8 · p. 28, Table 14 · p. 28

4.9 COMPARISON ON FLORES BENCHMARK

• On overlapping language pairs, the method’s distilled models yield higher SPBLEU than M2M-124 on Flores-101, though differing training data limit the comparison.The authors caution that public-benchmark comparisons are not necessarily meaningful because the models were not trained on the same data.Source: §4.9 · p. 28, Table 15 · p. 28

5.1 NON-ENGLISH-CENTRIC BRIDGING

• Bridging improves automatic scores in some cases, especially for Indian languages and close dialects, but most Native American and Bantu languages and languages without close relatives lose quality.Direct English translation has the highest CHRF in 19 of 28 cases; bridging appears to help when the intermediate language is both relatively high quality and close to the target.Source: §5.1 · pp. 29–30, Table 16 · p. 29

5.2 ZERO-SHOT TRANSLITERATION

• Zero-shot transliteration from Latinized Indian languages to native scripts worked well out of the box and appeared more robust than existing libraries to informal spellings.It could also correct spelling, but sometimes changed input content or hallucinated, and dialect could shift during transliteration.Source: §5.2 · p. 30

5.3 THE “PERIOD TRICK”

• Adding terminal punctuation produced small but consistent CHRF gains on distilled models, with all language pairs benefiting in the xx→en direction.The comparison covers evaluation sets with and without terminal punctuation on 26-language distilled models.Source: §5.3 · p. 30, Table 17 · p. 31

5.4 ROBUSTNESS TO NON-STANDARD GLYPH USAGE

• The model was robust to many nonstandard Unicode glyph variants, but West African languages showed relatively large CHRF changes with nonstandard glyphs.The table comparison also reports drops when apostrophes are removed entirely.Source: §5.4 · p. 31, Table 19 · p. 32

5.5 NON-UNICODE FONTS

• The authors encountered non-Unicode fonts in data for Ewe, Kashmiri, Meiteilon, Mooré, Navajo, and Tamazight, suggesting hidden data may exist in such fonts.The text describes ASCII input rendered through special fonts and mentions attempted reconstruction of some fonts.Source: §5.5 · pp. 31–32

• Native speakers identified reference-translation problems in cases with extremely low automatic metrics, including instances where model outputs appeared better than the references.Some examples had BLEU below 1.0 despite large, clean corpora.Source: §6 · p. 33
• Speakers helped identify politically sensitive script choices and transliterate text into the appropriate script.In one case, the initial script was associated with colonial times and had been replaced across the region.Source: §6 · p. 33
• Interviews found enthusiasm for lower-quality translation offerings among speakers consulted, while the authors caution that no individual represents an entire community.The paper says community members must determine whether imperfect models are useful or offensive, and notes differing opinions remain an open concern.Source: §6 · p. 33
• Community members clarified dialect production and language naming, including names that may be disliked or exclusive to some groups.They helped distinguish colonial or widely known names from preferred native names and identify the limits of labels such as “Manipuri.”Source: §6 · pp. 33–34

7.1 MAIN FINDINGS

• The authors built relatively clean web-mined monolingual datasets for over 1,000 languages, including more than 1 million sentences for over 200 languages and over 100,000 sentences for more than 400.They highlight semi-supervised LangID, document-level consistency, and word-based and custom filtering techniques.Source: §7.1 · p. 35
• Training on the mined dataset and parallel data spanning 112 languages produced multilingual models capable of translating across 1,000 languages.The authors report benefits from model capacity, increasing the number of self-supervised languages, back-translation, and self-training.Source: §7.1 · p. 35
• The authors evaluate on 38 languages, emphasize CHRF for long-tail evaluation, and provide approximate reference-free scores and human evaluations for distilled models.They report that high-quality practical MT models for long-tail languages are possible with their approach.Source: §7.1 · p. 35
• Qualitative and quantitative analyses identify errors involving distributionally similar and infrequent tokens, plus verbose and inaccurate outputs for short or single-word queries.The paper links these error patterns to extreme data sparsity in the zero-resource setting and also reports experiments on bridging, transliteration, punctuation, and glyph robustness.Source: §7.1 · p. 35
• Native speakers helped evaluate, understand, filter, and improve datasets and models, and contextualize how the systems fit their communities.Source: §7.1 · p. 35

7.2 RELATED WORK

• The work extends prior multilingual supervised, zero-resource, and self-supervised MT by scaling model size and language coverage and adding self-training and round-trip consistency filtering.The related-work discussion also notes that multilingual web datasets commonly have quality issues, particularly for low-resource languages.Source: §7.2 · p. 36

7.3 FUTURE WORK

• Future work includes dictionary grounding for distributionally similar words and combining web-mined multilingual resources with hand-curated, language-specific datasets and tools.The authors describe these approaches as complementary and suggest community contributions such as specialized filters and orthography or script tools.Source: §7.3 · p. 37
• Joint speech-and-text models are proposed as important for addressing data sparsity among languages with little web text or no standardized writing systems.Source: §7.3 · p. 38

• Human ratings use a seven-point scale from nonsense to perfect, alongside CHRF scores for the distilled models.The authors caution that human evaluation numbers are difficult to compare across languages.Source: A COMPLETE HUMAN EVALUATION AND CHRF… · p. 51

• Audits rated 100-sentence samples for 72 languages, with the breakdown reported in Table 22.Source: B COMPLETE AUDIT RESULTS · p. 51, Table 22 · p. 52
• Table 22 reports the percentage of each audited sample assigned to each label.Source: Table 22 · p. 52
• The combined score gives an approximate notion of the share of data considered usable.Source: Table 22 · p. 52

• Unlike the loose version, strict RTTLANGIDCHRF penalizes outputs in the wrong language.Source: §B.1 · p. 52
• Strict RTTLANGIDCHRF shows a clear trend and an apparent upper bound in quality when plotted against log data size.Source: §B.1 · p. 52
• Strict RTTLANGIDCHRF correlates worse with CHRF and data size than the loose version.Source: §B.1 · p. 52

• The Bengali-script corpus was converted because online Meeteilon in Meetei Mayek was scarce, likely in part because non-Unicode text was inaccessible to the LangID model.Source: C TRANSLITERATION FOR MEITEILON · p. 52
• Figure 4 depicts four composed WFST components: normalization, Bengali-to-Latin mapping, post-processing, and reverse romanization.The workflow is T = N ◦ R′ ◦ P ◦ R−1; the Bengali-to-Latin stage adds placeholders for later processing.Source: Figure 4 · p. 53, C TRANSLITERATION FOR MEITEILON · p. 54
• Meetei Mayek has a near-bijective grapheme-to-phoneme mapping, while Bengali-script conventions allow multiple letters or clusters for one Meiteilon sound.Source: C TRANSLITERATION FOR MEITEILON · p. 53
• Bengali orthography is ambiguous for Meiteilon, so transliteration to Meetei Mayek requires a many-to-one mapping.Source: C TRANSLITERATION FOR MEITEILON · p. 53
• Context-dependent post-processing resolves placeholders, converting Bengali virama to lonsum iyek for designated consonants and otherwise to apun iyek.Source: C TRANSLITERATION FOR MEITEILON · p. 54

• The mappings were found by copying text from an environment where it rendered into one where the font was unavailable, then identifying the matching Unicode character.Source: D TABLES OF NON-UNICODE FONTS · p. 54
• The authors distinguish their Meiteilon-specific Bengali-to-Latin operation R′ from Nisaba’s more general reversible Bengali and Assamese romanization R.Source: D TABLES OF NON-UNICODE FONTS · p. 54
• Table 23 gives ASCII-to-Unicode mappings for three possibly incomplete fonts used by three African languages with extended Latin characters.Source: Table 23 · p. 56

E LIST OF LANGUAGES WITH REGIONS, APPROXIMATE SPEAKER COUNTS

• The list records each language’s name, estimated L1 speaker count, continent, script, cluster, SSLID language-identification F1, and RTTLANGIDCHRF score where available.Speaker estimates follow van Esch et al. (2022).Source: E LIST OF LANGUAGES WITH REGIONS,… · p. 57
• RTTLANGIDCHRF is described as an approximate measure of model quality and should not be trusted as a reliable measure of translation quality.Source: E LIST OF LANGUAGES WITH REGIONS,… · p. 57
• The English and Spanish entries list estimated speaker counts of 550M and 490M, respectively.Source: E LIST OF LANGUAGES WITH REGIONS,… · p. 57
• The list pairs language codes and names with approximate speaker counts, region, script, clustering, F1, and RTT fields.These column labels appear in the table header, alongside entries such as Malay and Tabasaran.Source: E LIST OF LANGUAGES WITH REGIONS,… · p. 64
• Entries include languages from Africa, the Americas, Asia, Europe, and Oceania, represented in scripts including Latin, Arabic, Cyrillic, Devanagari, and Ethiopic.For example, the list includes Fang, Shuar, Tabasaran, Kumaoni, and Male (Ethiopia).Source: E LIST OF LANGUAGES WITH REGIONS,… · p. 64
• Speaker estimates in this part range from 5K for several listed languages to 80M for Malay.The table also reports language-specific clustering, F1, and RTT values where present; some entries show a dash for RTT.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 64–65
• The listed languages span the Americas, Africa, Asia, and Oceania, with occasional entries from Europe.Examples include Southern Kurdish in Asia, Mende in Africa, and languages from the Americas and Oceania.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 68, 69, 71
• Each entry pairs a BCP-47 code and language name with monolingual-data size, approximate speaker count, continent, script, cluster, F1, and RTT fields.The table header appears in the language-list passages, while the entries provide examples of these fields.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 69, 71, 72
• Approximate speaker counts range from 5K for many listed languages to tens of millions for entries such as Hakka Chinese.The entries include examples from 5K through 30M speakers.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 70–71
• Each entry pairs a BCP-47 code and language name with speaker count, region, script, clustering F1, and RTT fields.The fields are shown in the table header, and the rows provide language-specific values, sometimes with RTT marked as unavailable.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 74, 75, 77
• The inventory includes languages from Africa, Asia, the Americas, Europe, and Oceania, with scripts including Latin, Arabic, Devanagari, Ethiopic, Syriac, N’Ko, and Bengali.Examples include Rumelian Turkish in Europe, Santali in Devanagari, Bagri in Arabic, and Bambara in N’Ko.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 75–77
• 14M is the listed approximate speaker count for Bambara, while many entries in this portion list counts of 5K or 50K.The table also includes larger counts such as 10M for Min Bei Chinese and 6M for Santali and Lambadi.Source: E LIST OF LANGUAGES WITH REGIONS,… · pp. 74–77