Omnilingual MT: Machine Translation for 1,600 Languages

Data

• The system combines large public multilingual corpora with human-curated and synthetic resources to expand coverage across long-tail domains, registers, and underdocumented languages.Its data strategy includes manually curated seed datasets and synthetic backtranslation.Source: §1 · p. 4

Model design

• Omnilingual MT specializes LLMs as either decoder-only models or encoder–decoder systems, using continual pretraining, retrieval augmentation, aligned encoders, non-parallel data, and expanded vocabulary support.Both approaches use a 256K-token vocabulary and improved pre-tokenization for underserved scripts.Source: §1 · p. 4

Evaluation and resources

• The work introduces a broad evaluation suite combining MetricX and ChrF with BLASER 3, OmniTOX, BOUQuET, and Met-BOUQuET, and releases the datasets and leaderboard freely.The resources support reference-free quality estimation, toxicity assessment, and scalable multilingual evaluation.Source: §1 · pp. 4–5

Results

• More than 1,600 languages are supported, with non-trivial translation performance from 1,600 and into about 1,200 languages, outperforming competing systems by a large margin.The system establishes new, often first, state-of-the-art results for most of the evaluated languages.Source: §1 · p. 4
• Omnilingual MT doubles the number of languages modern models understand sufficiently well, from about 200 to over 400, while addressing baselines’ failure to generate undersupported languages coherently.Evaluation on English-to-1,560 Bible translations shows that Omnilingual MT substantially widens the set of languages with coherent generation.Source: §1 · pp. 4–5

Efficiency

• Specialized 1B to 8B parameter models match or exceed the MT performance of a 70B-parameter LLM baseline, yielding a favorable efficiency–performance tradeoff for low-compute settings.The comparison supports specialization rather than scale as a more reliable path to high-quality multilingual translation.Source: §1 · p. 4

Prior multilingual MT

• NLLB established strong translation quality for 200 languages, while setting FLoRes-200 as a de facto evaluation standard.Its data curation, architecture, and optimization strategies challenged the assumption that multilingual scaling must sacrifice quality.Source: §2 · p. 6

Coverage limitations

• Most systems still omit the vast majority of the world’s 7,000 languages, and often understand underserved languages through transfer without generating them faithfully.This generation bottleneck particularly limits practical support for long-tail languages.Source: §2 · p. 6

Data and linguistic challenges

• Extreme multilingual scaling is constrained by typological diversity, uneven corpora, and underrepresentation of dialects, colloquial styles, and marginalized languages.Synthetic data, bitext mining, and multilingual transfer have helped extend coverage, but larger datasets alone do not ensure equitable support.Source: §2 · p. 6

Evaluation challenges

• Evaluation remains a bottleneck beyond 200–300 languages because existing benchmarks lack broader coverage and metrics may fail to capture meaning or require typological calibration.At 1,000+ languages, fluent-looking outputs may be unintelligible or unrelated to the target language, requiring quality-estimation frameworks that preserve metric fidelity.Source: §2 · pp. 6–7

Paper contributions

• Omnilingual MT addresses these structural challenges by targeting broader and deeper coverage, efficient smaller-scale models, robust long-tail generation, and evaluation frameworks for 1,600+ languages.The paper also introduces XSTS+R+P, Met-BOUQuET, BLASER 3, and OmniTOX as contributions toward Omnilingual evaluation.Source: §2 · p. 7

3.1 Referring to languages

• Languages are counted as ISO 639-3 and ISO 15924 language–script combinations, with Glottolog codes available when finer granularity is needed.The paper notes that ISO classification is complex and may disagree with speakers’ conceptualizations; Mandarin examples distinguish traditional and simplified Han scripts.Source: §3.1 · pp. 7–8

3.2 Quality translations from or into underserved languages

• High-quality underserved-language translation requires addressing both missing parallel or evaluation resources and limited access to proficient translators or bilingual speakers.The authors recommend pivoting through high-resource languages familiar to speakers, rather than assuming English is always optimal, and supplying paragraph-level domain, participants, register, and tone context.Source: §3.2 · pp. 8–9

3.3 Resource levels

• Translation quality shifts above 1M and approximately 40K parallel documents, motivating resource distinctions based on parallel-data availability.Figure 3.1 measures OMT-LLaMA Bible-benchmark mean xCOMET across 1,560 languages and shows the global correlation between quality and primary-source parallel documents.Source: §3.3 · p. 9, Figure 3.1 · p. 10
• The final resource scheme labels languages high-resource above 50M parallel pairs, mid-resource above 1M, low-resource from 40K to 1M, extremely low-resource from 1K to 40K, and zero-resource below 1K.The thresholds are intended to align resource levels with translation quality, while anomalies reflect transfer, poor or narrow data, tokenizer coverage, and language-code misattribution.Source: §3.3 · pp. 9–10
• The resource classification remains imperfect: some languages below 1K documents perform well through similarity-based transfer, while others with over 1M documents perform poorly because of data or representation problems.The paper also identifies narrow domains, rare scripts, tokenizer limitations, and loosely defined language boundaries as possible explanations.Source: §3.3 · pp. 9–10

3.4 Describing languages in prompts

• Prompts describe target varieties in natural language, optionally specifying script, locale, or dialect, with names mapped primarily through Langcodes and Glottolog.Script descriptions are omitted for well-known varieties expected to use one dominant script, while dialect names disambiguate otherwise identical varieties in FLoRes+.Source: §3.4 · pp. 10–11

4 Creating High-Quality Datasets

• The dataset strategy combines existing multilingual corpora with new monolingual, aligned, synthetic, mined, and evaluation resources to address gaps in long-tail languages, domains, and registers.Continual pretraining draws on resources including Common Crawl-derived corpora, Bible data, PanLex, Tatoeba, CC-NLLB-200, OMT Primary, OMT Langwise, and LTPP, while evaluation uses Bible subsets and standard benchmarks such as FLoRes+.Source: §4 · p. 11, §4.1 · pp. 11–12

4.2 Synthetic Data for CPT

• Backtranslation and bitext mining extend parallel data where little or no parallel text exists, using monolingual corpora, MT systems, multilingual embeddings, and quality filtering.Backtranslation uses NLLB for its supported languages and an unrestricted LLaMA-based system for broader coverage; mining uses global alignment in OmniSONAR with FAISS-based nearest-neighbor search.Source: §4.2 · p. 13, §4.2.1 · pp. 13–14, §4.2.2 · p. 16

4.2.1 Backtranslation

• A 1:9 or 1:4 backtranslation-to-natural-bitext ratio maximizes performance for lower-resource languages, while other resource buckets improve as backtranslation increases.The resulting backtranslation dataset contains roughly 270 million sentences spanning more than 2,000 languoids, with 51% of data from low-resource, 26% from very-low-resource, and 23% from high-resource languoids.Source: §4.2.1 · pp. 14–15

4.2.2 Bitext Mining

• Adding mined alignments improves FLoRes+ ChrF++ across high- and low-resource settings, including relative gains of 2.95% for Greek, 2.74% for Turkish, and 5.12% for N’Ko.The controlled comparison uses LLaMA3.2 3B Instruct with fixed batch size, training steps, and other hyperparameters.Source: §4.2.2 · p. 16, Table 4.5 · p. 16

4.2.3 Conclusions and limitations

• Synthetic-data production remains limited by incomplete Common Crawl coverage and expensive resource-hungry models, while iterative generation, improved filtering, and non-English pivot mining are identified as future directions.The authors emphasize that the current data collection is not exhaustive and that scaling synthetic-data methods is costly.Source: §4.1 · p. 13, §4.2.3 · pp. 16–17

4.3 Seed Data for Post-Training: MeDLEy

• MeDLEy supplies professionally translated, manually constructed data for 109 low-resource languages, combining five source languages, eight pivot languages, five domains, and 61 cross-linguistic grammatical features.MeDLEy-source contains 605 paragraphs, roughly 2,200 sentences, and 34K English-counted words; its paragraphs are designed for accessible translation by lay community translators.Source: §4.3 · p. 17, Figure 4.1 · p. 17

4.3.2 Approach

• MeDLEy constructs grammatically diverse parallel data by transferring cross-linguistically shared grammatical functions from source corpora into arbitrary target languages.The framework assumes that grammatical functions often survive translation even when their surface forms differ across languages.Source: §4.3.2 · pp. 19–20
• The corpus combines grammatical-feature curation, five source languages, five domains, expert-written source paragraphs, eight pivot languages, and professional translation into selected low-resource targets.Target languages were selected using translator availability, prior coverage gaps, and language-family diversity.Source: §4.3.2 · p. 20

4.3.3 Experiments

• MeDLEy matches or outperforms baseline datasets with equal token budgets, improves into-English translation, and yields generally modest gains when added to existing datasets.These trends also hold for NLLB-Seed and evaluation metrics including xCOMET and MetricX.Source: §4.3.3 · pp. 20–21, Table 4.7 · p. 21

4.3.4 Conclusions and limitations

• MeDLEy spans 109 low-resource languages and provides broader grammatical diversity and greater fine-tuning impact than pre-existing seed datasets, while remaining limited by budget, language-specific transfer, evaluation scarcity, and vendor quality risks.Its design emphasizes common cross-lingual functions rather than language-specific phenomena, and inaccurate translations may occur more often in low-resource languages.Source: §4.3.4 · pp. 21–22

4.4 Evaluation Data

• The evaluation suite combines BOUQuET, a reserved Bible partition, and FLoRes+, because no single benchmark provides both broad language coverage and varied domains.The Bible offers long-tail coverage, whereas BOUQuET and FLoRes+ provide more varied domains and resource levels.Source: §4.4 · p. 22, §4.4.3 · p. 24

4.4.1 BOUQuET

• BOUQuET is created from scratch by linguists in eight non-English languages with gold English translations, contextual information, register labels, paragraph structure, and eight previously underrepresented domains.Its design addresses narrow domain coverage, contamination, English-centric difficulty, and missing translation context in earlier FLoRes-derived datasets.Source: §4.4.1 · p. 22
• BOUQuET expands to 275 languages spanning 56 language families and 33 scripts, with community and vendor contributions; French, Indonesian, Swahili, and Spanish pivots ease resourcing more than German and Hindi.English-only vendor workflows produce lower-quality translations, higher costs, and less completed work.Source: §4.4.1 · p. 23, Figure 4.2 · p. 23

4.4.2 Bible Evaluation Partition

• The Bible evaluation partition uses the Gospel of John as a multi-way test set for 1,561 language varieties, despite likely training contamination, while the three benchmarks collectively cover over 1,700 varieties or over 1,600 unique languages.FLoRes-HRL and FLoRes-Hard provide specialized subsets for higher-resource and particularly low-performing lower-resource languages.Source: §4.4.2 · p. 23, §4.4.3 · p. 24

5.1 Motivation and Approaches

• The work specializes general-purpose decoder-only LLMs either by direct translation fine-tuning or by deriving an encoder-decoder Transformer from the LLM.These strategies correspond to the OMT-LLaMA and OMT-NLLB approaches explored in Sections 6 and 7.Source: §5.1 · p. 24

5.2 Vocabulary Extension and Tokenization

• OMT-NLLB uses separate 256K-token encoder and decoder vocabularies, while OMT-LLaMA extends the original LLaMA3 vocabulary with tokens for 1,500+ languages.The OMT-NLLB input tokenizer is trained from scratch for over 1.5K languages, whereas its output tokenizer extends LLaMA3 for 200 languages.Source: §5.2 · p. 25
• The tokenizer adaptations modify pre-tokenization and expand LLaMA3 from 128K to 256K tokens, reducing average fertility from 80.7 to 44.8 tokens per sentence across 212 FLORES+ languages.The method uses continued BPE merging over balanced multilingual parallel and web data, with additional Unicode and diacritic-aware splitting patterns.Source: §5.2 · p. 25
• A 256K-token vocabulary improved relative ChrF++ by 26% for out-of-English (17.8 →22.5) and 7% for into-English (35.9 →38.7) on FLoRes+.The ablation continuously pretrained LLaMA3.2 1B Instruct models for 30K steps with identical data and hyperparameters, finding gains across all language-resource levels.Source: §5.2 · p. 26, Table 5.1 · p. 26

6.1 Base models

• The main OMT-LLaMA is an 8B LLaMA 3.1 Instruct model modified with a multilingual tokenizer and expanded input and output embeddings, alongside 1B and 3B variants.The smaller models use LLaMA 3.2 Instruct initialization and follow the same vocabulary adaptation and training recipe.Source: §6.1 · p. 26

6.2 Continued Pretraining

• Continual pretraining combines monolingual language modeling with parallel-document translation to specialize the base models for multilingual machine translation.Language names precede monolingual documents, while translation prompts specify source and target languages.Source: §6.2 · p. 26

6.3.1 Supervised Fine-Tuning

• SFT combines instruction-following and translation data, using approximately 600k examples across 10 languages and supplementing them with SMOL and MeDLEy translation datasets.The base mixture includes conversational instruction-following, machine translation, evaluation, automatic post-editing, and other language tasks.Source: §6.3.1 · p. 27

6.3.2 Reinforcement Learning

• DAPO addresses RL optimization difficulties by using group size N = 64, asymmetric clipping, dynamic sampling, KL regularization, and a balanced 50/50 ChrF++–MetricX reward.Diverse instruction data was critical for stable, generalizable gains, whereas narrowly templated data improved only in-distribution performance.Source: §6.3.2 · pp. 27–28

6.3.3 Results

• SFT consistently improves CPT across benchmarks, including full-set FLoRes+ English-to-other directions from 0.609 to 0.656 and BOUQuET paragraph-level →en from 0.527 to 0.549.SFT also improves the hardest FLoRes+ →en subset from 0.376 to 0.382 and targets languages represented during post-training.Source: §6.3.3 · p. 28, Table 6.1 · p. 28
• RL further improves SFT on nearly every evaluated direction, raising BOUQuET sentence-level →en from 0.732 to 0.741 and paragraph-level →en from 0.549 to 0.555.On FLoRes+, RL raises →en(hard) from 0.382 to 0.393 and does not degrade any evaluated direction.Source: §6.3.3 · p. 28, Table 6.1 · p. 28

6.4.1 Algorithm Overview

• RAG retrieves similar parallel source-target pairs through full-text and vector search, filters and reranks candidates using similarity and quality signals, and inserts up to 80 examples into prompts.When direction-specific data is sparse, language-agnostic embeddings enable on-the-fly mining of target examples with cosine similarity above 0.7.Source: §6.4.1 · p. 29

6.4.2 Experiments and Results

• RAG consistently improves all automatic metrics for LLaMA3.1-8B, LLaMA3.3-70B, and OMT-LLaMA-8B, with larger gains when more examples are available and stronger sentence-level improvements.The evaluation covers 56 BOUQuET directions, including directions with fewer than 30k examples and directions using on-the-fly mining.Source: §6.4.2 · p. 29, Table 6.2 · p. 30

7 Encoder-Decoder Modeling

• OMT-NLLB combines a cross-lingually aligned encoder with a standard encoder–decoder Transformer to translate from 1,600 source languages into 250 target languages.The model uses 3B parameters and is designed to exploit both parallel and non-parallel data before recovering a sequence-to-sequence MT model without a pooled sentence bottleneck.Source: §7 · p. 31
• The method removes OmniSONAR’s pooled sentence representation bottleneck by introducing full-sequence encoder–decoder attention after initial aligned-encoder training.The initial stage uses parallel translation and monolingual autoencoding data, while later stages operate on parallel data after pooling is removed.Source: §7 · p. 31

7.1 Leveraging an Aligned Encoder for Enhanced Decoder Training

• Autoencoding monolingual data improves decoder quality beyond parallel-only training, especially for low-resource languages with limited parallel corpora.Performance improved in 105 of 114 languages receiving autoencoding data, with an average gain of 5.20 chrF++ points, although overall-model gains were smaller, consistent with multilinguality trade-offs.Source: §7.1 · p. 32

7.2 Decoder Warm-up for Token-Level Cross-Attention

• Decoder warm-up with token-level cross-attention yields further gains across all benchmarks and is necessary for stable transition away from sentence-level pooling.Updating only the decoder first lets it adapt to full encoder sequences; skipping this phase causes exploding gradients during final training.Source: §7.2 · p. 33

7.3 End-to-End Parallel Fine-Tuning

• The final OMT-NLLB model achieves the strongest results across all evaluation sets while maintaining a compact 3B-parameter size.End-to-end fine-tuning jointly updates encoder and decoder parameters, enabling task-specific optimization of token-level interactions and improving substantially over the OmniSONAR baseline and preceding stages.Source: §7.3 · p. 34, Table 7.1 · p. 34

8.1 Human Evaluation Protocol: XSTS+R+P

• XSTS+R+P extends XSTS with sentence-level register scoring and paragraph-level penalties for contextual inconsistencies, while remaining easier to scale than more complex protocols such as MQM.It was designed to preserve sensitivity to register and paragraph context in multilingual evaluation.Source: §8.1 · pp. 34–35
• XSTS+R+P achieves a mean Krippendorff’s α of 0.80, substantially exceeding the baseline protocols and typical translation-evaluation agreement levels.The protocol also correlates positively with XSTS (0.65) and RSQM (0.62) using Kendall’s Tau.Source: §8.1 · p. 36
• XSTS+R+P scores diverge further below XSTS as source context length and social differential increase, especially for lower-to-higher-status address.Its score distribution is similar to XSTS but less concentrated at the upper end, indicating additional evaluation nuance.Source: §8.1 · p. 36

8.2 Met-BOUQuET

• Met-BOUQuET covers 161 language directions and 119 unique language varieties through two annotation rounds designed for multilingual, multi-way metric evaluation.The benchmark includes diverse language families, regions, resource levels, registers, and domains, with 73% of directions excluding English.Source: §8.2 · pp. 37–39
• Round 2 of Met-BOUQuET yields approximately 49K annotated translation pairs, with 57% receiving different consensus scores and thus expressing an induced human preference.The round compares strong OMT-LLaMA-based systems with strong external baselines across challenging directions.Source: §8.2 · pp. 38–39

8.3 BLASER 3

• BLASER 3 addresses the limited multilinguality, zero-shot generation, and single-protocol constraints of existing reference-free QE metrics by combining OmniSONAR embeddings, multi-task protocol training, and real plus synthetic data.Its design targets generalization to thousands of languages and coverage of hundreds of translation directions.Source: §8.3 · pp. 39–40

8.3.3 Experimental Setup

• BLASER 3 evaluation uses 1.6M examples across six protocols and 204 directions, with normalized scores, deduplication, contamination removal, and protocol-specific regression heads.Models are selected using Met-BOUQuET validation performance and evaluated with Spearman’s ρ on the corresponding test heads.Source: §8.3.3 · pp. 41–42

8.3.4 Results

• BLASER 3 surpasses prior multilingual QE models on Met-BOUQuET, gaining +0.08 over MetricX-24 and +0.12 over xCOMET-XL, with improvements attributed mainly to the non-English-source direction.Synthetic data further improves performance, supporting expanded domain and language coverage.Source: §8.3.4 · p. 42

8.4.1 Automatic analysis

• Combining source and reference information generally outperforms either alone, while GlotLID adjustment benefits reference-free and most reference-based metrics by detecting wrong-target-language translations.The recommended metrics are LID-adjusted BLASER 3 without references, and LID-adjusted MetricX or xCOMET using both source and reference.Source: §8.4.1 · p. 44
• Across resource-level groups, BLASER 3 outperforms every other metric on average, while low-resource source and especially target languages make automatic evaluation more difficult.LID adjustment helps translations into lower-resource languages, and reference-based metrics are strongest in several low-resource settings.Source: §8.4.1 · p. 44, Table 8.6 · p. 45

8.4.2 Manual analysis: automatic metrics vs human judgment

• Automatic metrics usually align with human judgments for obvious hallucinations and opposite meanings, but miss paragraph-level context, salient-word importance, register, and the inadequacy of literal translations.The manual analysis covers Swedish-English and Italian-Romanian directions and finds register problems especially apparent when translating out of English.Source: §8.4.2 · pp. 45–46

8.5 OmniTOX

• OmniTOX extends MuTox-style toxicity detection from 200 to 1,600+ languages by replacing SONAR with OmniSONAR while retaining a simple MLP classifier over frozen cross-lingual embeddings.The classifier uses a two-layer MLP with dropout and is designed to support zero-shot transfer to languages unseen during training.Source: §8.5.1 · pp. 46–47

8.5.3 Results

• Baseline ZS, trained exclusively on English and Spanish, achieves 0.793 overall ROC AUC and 0.821 mean per-language ROC AUC, outperforming MuTox despite using 28 fewer training languages.OmniTOX outperforms MuTox on 28 of 30 languages, improving mean per-language ROC AUC from 0.798 to 0.860.Source: §8.5.3 · pp. 48–49
• OmniTOX achieves an overall ROC AUC of 0.845, outperforming MuTox by +0.058, with the embedding upgrade contributing +0.052 and classifier optimization adding +0.006.The embedding change accounts for 90% of the total improvement, indicating that representation quality drives most of the gain.Source: §8.5.3 · p. 48, Table 8.9 · p. 49
• OmniTOX’s improvements over MuTox are primarily driven by upgrading from SONAR to OmniSONAR embeddings, while Baseline ZS shows strong zero-shot generalization to 1600+ languages.The experiments suggest OmniSONAR’s cross-lingual representations can generalize beyond the 30 training languages.Source: §8.5.3 · p. 49
• OmniTOX inherits OmniSONAR’s weaknesses, including potentially weaker representations for languages with limited pre-training data and limited preservation of toxicity-specific semantic distinctions.These weaknesses may constrain fine-grained toxicity detection in some languages.Source: §8.5.3 · p. 49
• Evaluation covered only 30 of OmniSONAR’s 1600+ supported languages, leaving performance on the remaining 1570+ languages dependent on unvalidated zero-shot transfer.Training data is also imbalanced, with English and Spanish comprising approximately 40% of examples and potentially biasing the model toward Western toxicity norms.Source: §8.5.3 · pp. 49–50
• Annotation subjectivity and cultural dependence may introduce annotator bias and underrepresent culturally specific toxicity patterns.Costa-jussà et al. (2024a) is cited in the supplied passage.Source: §8.5.3 · p. 50
• Because OmniTOX performs sentence-level binary classification, it may miss context-dependent toxicity and cannot represent toxicity severity or type.These limitations may affect downstream applications requiring fine-grained toxicity analysis.Source: §8.5.3 · p. 50

9.1.2 Evaluating with standard benchmarks

• OMT systems match strong baselines for high-resource translation and significantly outperform them on mid- and low-resource directions across BOUQuET and FLoRes+.On BOUQuET, OMT-NLLB is preferable for high- and mid-resource targets, while OMT-LLaMA excels for low- and very-low-resource targets; FLoRes+ shows the same broad pattern.Source: §9.1.2 · p. 51, Table 9.2 · p. 51, Table 9.3 · p. 52

9.1.3 Evaluating long-tail understanding and generation

• OMT-LLaMA 8B strictly outperforms all baselines in 1,045 Bible languages, while OMT-NLLB 3B consistently outperforms all baselines across the 1,600 evaluated languages.These relative-understanding results are reported for ChrF++ and xCOMET, with similar findings for MetricX and BLASER 3.Source: §9.1.3 · p. 53, Figure 9.2 · p. 54
• OMT models substantially expand long-tail coverage: OMT-LLaMA 8B and OMT-NLLB understand 440 and 416 Bible languages above the passable threshold, versus 221 for NLLB-200.For generation, baseline quality becomes near-random at about 300–400 languages, whereas OMT quality holds for about 1,200; automatic metrics cannot fully assess grammaticality and fluency.Source: §9.1.3 · pp. 55–56, Figure 9.4 · p. 55, Figure 9.5 · p. 56

9.1.4 Comparing across model sizes and architectures

• OMT-NLLB best translates from low- and zero-resourced languages, whereas OMT-LLaMA is competitive for higher-resourced inputs and generates over 1,000 languages versus “only” 250 for OMT-NLLB.The 1B OMT-LLaMA model slightly underperforms larger variants for most languages, especially the hardest ones. Differences between OMT-LLaMA and OMT-NLLB may stem from architecture, training tasks, training-data composition, training intensity, or their combination.Source: §9.1.4 · pp. 57–58

9.2 Human Evaluation

• Human evaluation favors OMT-LLaMA on 44 of 57 directions (77%), with mean XSTS+R+P improving from 2.67 for baselines to 3.38 for OMT.OMT-LLaMA outperforms baselines in every direction group and never trails by more than 0.3 points on average, though results remain below the “really good” score of 4.Source: §9.2 · pp. 58–59, Table 9.6 · p. 58, Figure 9.8 · p. 59

9.3 Added Toxicity Automatic Evaluation

• None of the evaluated systems show meaningful added toxicity, with flagging rates below 1.5% across 207 BOUQuET languages.A small directional asymmetry aligns with architecture: encoder-decoder models are slightly higher for X→Eng, whereas decoder-only models are slightly higher for Eng→X.Source: Table 9.6 · p. 58, §9.3 · p. 60
• The evaluation measures average added toxicity and flagging rate (%) for 207 languages in BOUQuET.Source: Table 9.7 · p. 60

Results

• Fine-tuning improves out-of-English translation but harms into-English performance, while retrieval augmentation almost always improves both models and directions.The retrieval exception is translation into the five mid-resourced languages; the fine-tuning degradation is hypothesized to reflect repetitive English outputs.Source: Table 10.1 · p. 61, §10 · pp. 61–62

Languages and Data

• The experiments target 25 selected languages—20 low- or very-low-resourced and 5 mid-resourced—using primary parallel data for translation into and out of English.Selected languages had low BOUQuET performance or were moderately difficult, and were available in the contemporaneous MeDLEy version; per-language data ranged from 300K to 11K examples.Source: §10 · p. 61

Conclusions

• Combining focused fine-tuning with retrieval augmentation improves targeted translation, with fine-tuning strongest for difficult-language outputs and retrieval strongest for inputs from those languages.The two techniques’ positive effects add up, although their effects differ by translation direction.Source: §10 · p. 62

Comparison with LLaMA-base

• Even after instruction fine-tuning and retrieval augmentation, LLaMA-base does not outperform unadapted OMT-LLaMA, establishing OMT-LLaMA as the preferable adaptation base.LLaMA-base generally shows similar qualitative trends, but unlike OMT-LLaMA does not exhibit fine-tuning’s negative effect on into-English translation.Source: §10 · p. 62

• Scaling beyond 1,000 languages required coordinated advances in multilingual data, vocabulary, architectures, and evaluation, including MeDLEy, BOUQuET, OMT-LLaMA, and OMT-NLLB.OMT-NLLB uses a three-stage strategy that exploits non-parallel data, while OMT-LLaMA makes minimal changes to standard language-model training.Source: §11 · p. 62
• 1B–8B-parameter specialized MT models match or exceed a 70B LLM, while substantially expanding coherent generation across 1,600 languages.Existing systems often interpret undersupported languages without generating them faithfully; Omnilingual MT improves cross-lingual transfer and generation feasibility.Source: §11 · p. 62
• Targeted finetuning and retrieval-augmented generation provide further quality gains when additional data exists for the relevant languages and domains.Source: §11 · p. 62
• Post-training multilingual extension cannot replace the larger gains from massively multilingual data and vocabulary introduced earlier in training.The findings motivate models that are multilingual by design and prepared to extend support to additional languages.Source: §11 · p. 62
• Public BOUQuET and Met-BOUQuET enable reproducible multilingual evaluation, while released data guidelines and OMT recipes support future inclusive models.The authors frame 1,600-language coverage as a baseline for continued work in translation, multilingual LLMs, and speech-to-text.Source: §11 · pp. 62–63

Data

• The named contributors led MeDLEy and backtranslation data work, OMT-NLLB development, continual-pretraining and retrieval experiments, OMT-LLaMA post-training, and BLASER 3 pipeline engineering.These contributions span data, modeling, and technical development.Source: Data · p. 64, Modeling · p. 64

Evaluation

• Evaluation contributors created and analyzed BOUQuET, Met-BOUQuET, XSTS+R+P, and OmniTOX, while coordinating manual translation quality, annotations, language-service relationships, and linguistic coverage.The evaluation work included data creation, validation, quality analysis, and feature-retention analysis.Source: Evaluation · p. 64

Project management

• Project and technical leadership coordinated partnerships, resourcing, engineering, model-family strategy, and the overall evaluation direction across the Omnilingual MT effort.Leadership roles covered program management, research direction, continual pretraining, and evaluation initiatives.Source: Project management · p. 64, Technical leadership · p. 64

A.1 More details on the approach

• MeDLEy was designed to provide domain- and grammatically diverse bitext across many languages, enabling future community extension without language-specific linguistic expertise.Its framework transfers grammatical diversity through translation while prioritizing accessible, natural source paragraphs.Source: §A.1 · p. 81
• The dataset uses five domains, five source languages, grammatical-feature templates, eight pivot languages, and professional translation into 109 low-resource target languages.Source paragraphs are manually parallelized through the pivots before commissioned translation into selected target languages.Source: §A.1 · p. 81

A.2 More details on data creation

• MeDLEy creation comprised batches of 254 and 352 paragraphs, covering 18 grammatical categories and 61 features, with two features removed after poor transferability.Templates randomly combine features, domains, source languages, and two-to-five sentence lengths, while later constraints improved feature-domain compatibility and naturalness.Source: §A.2 · p. 82

A.3 Features

• The grammatical inventory spans case, number, tense, aspect, mood, evidentiality, politeness, voice, valency, negation, questions, subordination, information structure, anaphora, pronouns, coordination, compounds, and emphasis.Middle voice and suppletion were dropped in Batch 2.Source: §A.3 · pp. 82–83

A.4 Guidelines for linguists and translators

• Linguist guidelines prioritized naturalness while requiring template features where feasible, allowing dropped features or added sentences; only 21 of more than 2,500 feature instances were dropped.The guidelines also requested contextual information such as utterance setting and referent genders.Source: §A.4 · p. 84
• MeDLEy guidelines require context-aware translation and transcreation so target texts preserve meaning while adapting language-specific information such as inclusivity and gender.Transcreation may remove information that does not apply in the target language or add distinctions needed to preserve the original interpretation.Source: §A.4 · pp. 85–86

A.5 Annotator details

• MeDLEy-source was translated into 109 target languages by professional native-level translators, with contextual-adequacy instructions and no permitted AI or automatic translation tools.Translations were checked for formatting and quality by vendors and the authors, including paragraph boundaries, digits, emojis, quotations, and markup tags.Source: §A.5 · p. 86

A.6 Sentence-level annotations

• MeDLEy sentence alignments combine SaT boundary probabilities, SONAR cross-lingual token alignment, and semantic similarity to produce 1:1 source–target segments.The procedure uses bidirectional monotonic alignment, boundary refinement, and optional length constraints, while diagnostic scores flag potentially noisy pairs for filtering or manual inspection.Source: §A.6 · p. 87

A.7 Examples from our dataset

• Table A.3 illustrates dataset entries by showing their feature templates, source and translated texts, domains, contextual information, and translation routes.The examples document multistep routes such as Spanish to English to Indonesian to Batak.Source: §A.7 · pp. 87–88, Table A.3 · p. 89

A.8.1 Feature distribution analysis

• MeDLEy has the highest entropy in 5 of 9 English feature categories, often including rarer features such as first person, perfect aspect, and clausal negation.Higher entropy indicates a more balanced paradigm distribution; Table A.4 summarizes entropy across datasets.Source: Table A.4 · p. 90, §A.8.1 · p. 90
• The study labels grammatical features with a Stanza parser and heuristics, interpreting higher entropy as a more balanced distribution over each paradigm.The analysis covers paradigms including tense, aspect, and formality, with full feature distributions reported separately.Source: §A.8.1 · p. 90
• For Hindi, MeDLEy often has the highest entropy, with more balanced tense, verbal-valency, and negation-type distributions than Wikipedia-derived NLLB-Seed.The Hindi analysis uses automatically translated SmolSent and NLLB-Seed, so translation effects may influence the findings.Source: §A.8.1 · p. 90

A.8.2 Feature retention study

• The study evaluates 10 linguistic features across 120 Russian- and Spanish-created paragraphs, their English translations, and corresponding two-hop translations.Features span morphology, morphosyntax, and information structure, including case, tense, voice, mood, word order, compounding, intensification, and evidentiality.Source: §A.8.2 · p. 90
• The feature study is limited to languages shared across the relevant datasets with available parsers and linguistic expertise; English is the only such language.This restriction constrains the analysis to the intersection of eligible datasets, tooling, and expertise.Source: §A.8.2 · p. 90
• Most morphosyntactic features transfer at rates above 50%, except marked word order, and some features lost in one hop resurface in the next.Russian-English-Spanish examples show marked word order disappearing in English but reappearing in Spanish, likely because Russian and Spanish share pragmatic word-order uses.Source: §A.8.2 · p. 92
• Imperative mood is the most robust feature, transferring in every one- and two-hop translation, while verbal-morphology features generally show the highest retention.Spanish passives retain slightly less because pasiva refleja lacks a clear English equivalent and some passive readings map to marked Russian word order.Source: §A.8.2 · p. 92
• Case retention is idiosyncratic because Russian has morphological case, whereas English and Spanish primarily express instrumental and dative meanings through restricted constructions.Russian instrumental case covers agents and copular attributes that do not consistently trigger equivalent prepositions in English or Spanish.Source: §A.8.2 · p. 93
• Feature loss can redistribute into other rare features or reappear later, especially when source and final languages share a function absent from the intermediary language.The qualitative analysis also attributes variation to differences in which grammatical forms and functions languages encode.Source: §A.8.2 · pp. 93–94

A.9 More details on datasets and language statistics

• Experiments cover English-to and English-from directions for Bambara, Ganda, Mossi, Wolof, and Yoruba, the five languages represented across all seed and evaluation datasets.MeDLEy has longer examples, SmolSent and SmolDoc shorter sequences, and SmolDoc is much larger, motivating shared per-language token budgets.Source: §A.9 · p. 95
• Because seed datasets differ in example counts, token lengths, and total size, experiments sample each language under a fixed token budget set by its smallest seed dataset.The dataset comparisons are summarized in Figures A.1-A.3, covering examples, token-length distributions, and total tokens.Source: §A.9 · p. 95, Figure A.1 · p. 95, Figure A.2 · p. 96

Models and decoding

• The study compares NLLB-200-3.3B sequence-to-sequence translation with LLaMA-3.1-8B-Instruct autoregressive instruction-following translation.NLLB uses language-tagged inputs and beam search, while LLaMA uses a minimal translation prompt. [Grattafiori et al., 2024]Source: §A.10 · p. 97

Training protocol

• Models are fine-tuned and evaluated separately for each language pair and direction, covering both into-English and out-of-English translation.Training maximizes target-sequence log-likelihood with teacher forcing, and the best checkpoint is selected using held-out validation performance.Source: §A.10 · p. 97

Languages and data

• Experiments cover Bambara, Mossi, Wolof, Yoruba, and Ganda, using sentence-level chunks for long SmolDoc samples.These languages are covered by the baseline MeDLEy-109 and evaluation datasets; sentence alignments define the chunking.Source: §A.10 · p. 97

Input representations

• NLLB encodes language codes with reserved tokenizer tokens, whereas LLaMA uses instruction fine-tuning with a simple source-to-target prompt template.All evaluated languages are technically supported by NLLB because its tokenizer reserves language-code tokens.Source: §A.10 · p. 97

Systems and efficiency

• Training uses A100 GPUs and mixed precision, with dynamic padding for NLLB and packing, FlashAttention-2, and FSDP for LLaMA.Inference uses ctranslate2 for NLLB and vLLM for LLaMA; training hyperparameters are listed in Table A.9 and inference hyperparameters in Table A.10. [Micikevicius et al., 2017] [Dao, 2023] [Kwon et al., 2023]Source: §A.10 · pp. 97–98, Table A.9 · p. 98

A.11 More details on the experiment results

• Seed data improves performance, especially for LLaMA, while NLLB maintains higher baseline and fine-tuned scores and remains state-of-the-art for low-resource MT.The comparison favors smaller sequence-to-sequence models over LLM-based MT in this setting, consistent with previous findings.Source: §A.11 · p. 98
• MeDLEy matches or outperforms baseline datasets in token-controlled evaluations, gains in the into-English direction, and shows similar trends under xCOMET and MetricX.These results support using improved xx-en systems to generate synthetic data from monolingual low-resource-language data.Source: §A.11 · p. 98
• Despite these gains, scores remain generally low, especially for en-xx, and adding MeDLEy to existing seed datasets usually yields only small improvements.Manual collection of a few thousand sentences is therefore unlikely to produce major low-resource-language gains; NLLB-Seed also contains 54% difficult C1/C2 texts versus 10.41% in MeDLEy.Source: §A.11 · pp. 98–99

B.1 XSTS+R+P

• XSTS+R+P measures sentence-level meaning correspondence and register within paragraphs, aggregating ratings to estimate semantic equivalence between languages.Annotators compare source and target sentences in context and assign scores from 1 to 5, including lexical, grammatical, and usage-expression meaning.Source: §B.1 · p. 105, Rating guidelines · p. 107

Rating guidelines

• The rating guidelines define scores from non-equivalence to complete equivalence while treating meaning, register, omissions, additions, and dialect consistency as material quality dimensions.Minor typos, grammatical errors, capitalization, and punctuation are ignored when they do not affect understanding.Source: Rating guidelines · pp. 105–107
• The pilot used 210 Russian-Spanish and Spanish-Russian sentence pairs in 50 paragraphs, showing that annotators could incorporate register and paragraph context.The final guidelines were revised because annotators identified lexical mistranslations more often than grammatical non-equivalent meaning.Source: Rating guidelines · p. 107

B.2 Detailed Selection of MT outputs Met-BOUQuET Round 1

• Met-BOUQuET Round 1 selects one output for each of 1,358 BOUQuET development and test sentences to balance quality, systems, and translation-error types.Selection combines candidate systems and four metrics, caps probably good translations at 50%, samples metric-best and metric-worst outputs, then fills the remainder randomly with slight upsampling of underrepresented systems.Source: §B.2 · pp. 107–108

B.3 Comparison to other MT metrics evaluation datasets

• Table B.1 compares MT-metrics evaluation datasets using protocols, language coverage, language pairs, source sentences, evaluated systems, and domains.The comparison includes MLQE Fomicheva et al. (2022), IndicMT Eval Sai B et al. (2023), AmericasNLP (Task3) De Gibert et al. (2025), and NLLB NLLB Team et al. (2024).Source: Table B.1 · p. 108, §B.3 · p. 108
• Met-BOUQuET uniquely combines broad multilinguality with parallel evaluation through identifiers assigned across source languages, paragraphs, and sentences.This enables cross-lingual comparisons across many languages and translation directions rather than restricting parallelism to one source language.Source: §B.3 · p. 108
• Met-BOUQuET Round 1 contains 100% bidirectional pairs and 60% English-free directions, while Rounds 1+2 include 118 English-free directions versus NLLB’s 20.Round 1’s 60% corresponds to 62 directions without English.Source: §B.3 · p. 108
• Met-BOUQuET provides comprehensive sentence-level metadata, including domain, register, linguistic annotations, and the MT system that produced each output.Although it contains a single target output, it supports more granular and fine-grained performance benchmarking than the other reviewed datasets.Source: §B.3 · pp. 108–109

C Cards

• The work catalogs the automatic and human evaluation metrics and protocols used throughout its evaluation.Table C.1 provides the consolidated metrics/protocols reference.Source: Table C.1 · p. 111

D Languages at a glance

• The language inventory covers 275 languages in BOUQuET and 120 in Met-BOUQuET, alongside training MeDLEy and FLoRes+ community-effort coverage.Table D.1 organizes entries by ISO 639-3 code, ISO 15924 script, language family, and dataset availability.Source: D Languages at a glance · p. 112, Table D.1 · p. 119