PaperlayerView exact revision on arXiv

Source-linked AI summary

Omnilingual MT: Machine Translation for 1,600 Languages

Omnilingual MT Team, Belen Alastruey, Niyati Bafna, Andrea Caciolai, Kevin Heffernan, Artyom Kozhevnikov, Christophe Ropers, Eduardo Sánchez, Charles-Eric Saint-James, Ioannis Tsiamas, Xiang "Tony" Cao, Chierh Cheng, Joe Chuang, Paul-Ambroise Duquenne, Mark Duppenthaler, Nate Ekberg, Cynthia Gao, Pere Lluís Huguet Cabot, João Maria Janeiro, Jean Maillard, Gabriel Mejia Gonzalez, Holger Schwenk, Edan Toledo, Arina Turkatenko, Albert Ventayol-Boada, Rashel Moritz, Alexandre Mourachko, Surya Parimi, Mary Williamson, Shireen Yates, David Dale, Marta R. Costa-jussà

arXiv:2603.16309v3cs.CL

TL;DR

Existing MT systems cover only a fraction of the world’s languages and often understand underserved languages without generating them faithfully. Omnilingual MT combines expanded multilingual data, specialized model architectures, and scalable evaluation resources to support more than 1,600 languages, with smaller models matching or exceeding a 70B LLM baseline and improved coherent generation.

  • Problem

    MT coverage and evaluation remain limited relative to the world’s languages, while systems often interpret underserved languages without generating them with meaningful fidelity.

  • Method

    Omnilingual MT combines large public corpora with newly created resources, specializes LLMs in decoder-only and encoder–decoder architectures, and develops multilingual evaluation artifacts.

  • Results

    Omnilingual MT supports more than 1,600 languages; its 1B–8B specialized models match or exceed a 70B LLM baseline and improve coherent generation for undersupported languages.

  • Takeaways & Limitations

    The results show that broad MT coverage requires robust language generation alongside cross-lingual understanding, and that specialization can provide strong performance at smaller scales.

  • Takeaways & Limitations

    MeDLEy grammatical coverage is constrained by budget, language-specific transfer limits, scarce labeled evaluation data, and greater reliance on external vendors for low-resource languages.

Abstract

from arXiv · show

High-quality machine translation (MT) can scale to hundreds of languages, setting a high bar for multilingual systems. However, compared to the world's 7,000 languages, current systems still offer only limited coverage: about 200 languages on the target side, and maybe a few hundreds more on the source side, supported due to cross-lingual transfer. And even these numbers have been hard to evaluate due to the lack of reliable benchmarks and metrics. We present Omnilingual Machine Translation (OMT), the first MT system supporting more than 1,600 languages. This scale is enabled by a comprehensive data strategy that integrates large public multilingual corpora with newly created datasets, including manually curated MeDLEY bitext. We explore two ways of specializing a Large Language model (LLM) for machine translation: as a decoder-only model (OMT-LLaMA) or as a module in an encoder-decoder architecture (OMT-NLLB). Notably, all our 1B to 8B parameter models match or exceed the MT performance of a 70B LLM baseline, revealing a clear specialization advantage and enabling strong translation quality in low-compute settings. Moreover, our evaluation of English-to-1,600 translations further shows that while baseline models can interpret undersupported languages, they frequently fail to generate them with meaningful fidelity; OMT-LLaMA models substantially expand the set of languages for which coherent generation is feasible. Additionally, OMT models improve in cross-lingual transfer, being close to solving the "understanding" part of the puzzle in MT for the 1,600 evaluated. Our leaderboard and main human-created evaluation datasets (BOUQuET and Met-BOUQuET) are dynamically evolving towards Omnilinguality and freely available.

1 Introduction

Omnilingual MT extends benchmarked machine translation beyond 1,600 languages through expanded data resources, specialized architectures, and multilingual evaluation artifacts. Its models improve coverage, generation, and efficiency relative to existing systems and large general-purpose LLMs.

  • Contribution: Omnilingual MT supports more than 1,600 languages, the broadest coverage of any benchmarked MT system to date.Its data strategy combines large public corpora with human-curated and synthetic resources targeting long-tail gaps.
  • Modeling: The system specializes LLMs as either standalone decoder-only models or modules within encoder–decoder architectures.The decoder-only approach extends LLaMA3 with multilingual continual pretraining and retrieval-augmented inference; the encoder–decoder approach uses an aligned encoder.
  • Evaluation: Omnilingual MT combines MetricX and ChrF with BLASER 3, OmniTOX, BOUQuET, and Met-BOUQuET for multilingual evaluation.BOUQuET is manually extended across linguistic families, while Met-BOUQuET provides multilingual quality estimation at scale.
  • Results: Omnilingual MT expands sufficiently understood languages twofold, from about 200 to over 400, and translates from 1,600 and into about 1,200 languages.It outperforms competitive translation systems by a large margin and establishes new or first state-of-the-art results for most evaluated languages.
  • Results: Specialized 1B–8B models match or exceed the MT performance of a 70B-parameter LLM baseline.The reported efficiency–performance advantage supports strong translation in low-compute settings.
  • Implications: Omnilingual MT also broadens coherent generation for undersupported languages, while its evaluation resources and leaderboard are freely available.The paper frames robust generation, rather than understanding alone, as necessary for large-scale MT coverage.

2 Expanding Machine Translation

Scaling MT beyond the established 200-language frontier requires coordinated advances in data, modeling, generation, and evaluation. Omnilingual MT addresses these constraints with broader resources, specialized architectures, and evaluation tools designed for thousands of languages.

  • Motivation: NLLB demonstrated strong translation quality for 200 languages, but most of the world’s 7,000 languages remain absent from technological representation.Existing systems continue to face a coverage frontier and generation failures for underserved languages.
  • Motivation: Long-tail languages span diverse morphologies, scripts, and sociocultural contexts, exposing the brittleness of closed-coverage MT systems.Adding a language requires more than simply acquiring data.
  • Evaluation: Evaluation benchmarks generally cover only 200–300 languages, while reference-based and reference-free metrics face challenges at multilingual scale.These limitations make quality assessment difficult when outputs are fluent but unintelligible or unrelated to the target language.
  • Motivation: The paper argues that massively multilingual MT needs broader coverage, robust generation for underserved languages, efficient smaller models, and reliable long-tail evaluation.This reframes scaling as a systems problem spanning how models are built, trained, and evaluated.
  • Approach: Omnilingual MT contributes multilingual data resources, two model architectures, and expanded metrics and human-evaluation artifacts.The paper presents MeDLEy and BOUQuET bitext, BLASER 3, OmniTOX, and Met-BOUQuET among these contributions.

3 Languages

The paper treats language coverage as a resource- and representation-dependent problem rather than a simple language count. It defines practical language categories, develops data strategies for underserved languages, and links parallel-data scale with translation quality.

  • Language definition: The paper uses ISO 639-3-coded linguistic entities as language candidates while acknowledging disagreements in language classification.ISO 15924 script codes distinguish language–writing-system combinations, which are typically counted separately.
  • Resource concepts: Underserved languages refer to communities lacking access to the full range of language technologies, especially machine translation.The term is preferred alongside the more data-oriented industry label “low-resource languages.”
  • Data and evaluation: Low-quality translation into or out of underserved languages reflects shortages in both high-quality parallel data and useful evaluation annotations.A shared difficulty is commissioning proficient translators or bilingual speakers.
  • Data creation: Pivot-language workflows must account for speaker access and working conditions, especially amid intergenerational disruption in underserved-language communities.Older and younger generations may have different proficiency profiles across the underserved language and an official language.
  • Data creation: Contextual information helps translators resolve ambiguities involving grammatical features, register, formality, domain, protagonists, and tone.English pronouns and limited formality marking can leave gender, number, and register underspecified.
  • Resource levels: The resource-level definition is dynamic, and experiments identify qualitative translation-quality shifts above 1 million and approximately 40K parallel documents.The paper further subdivides resource levels beyond the conventional 1-million-document threshold.
  • Resource levels: Resource buckets contain anomalies: some languages with fewer than 1K documents perform well, while some languages with more than 1M documents perform poorly.The paper hypothesizes positive transfer, low-quality or narrow-domain data, tokenizer coverage, and language-attribution errors as possible explanations.

4 Creating High-Quality Datasets

The paper builds broad-coverage MT datasets by combining public corpora with curated, synthetic, mined, and manually created multilingual data. These resources expand grammatical and linguistic coverage, improve low-resource translation, and support evaluation across more than 1,600 languages, while remaining constrained by data availability, cost, and quality-assurance limits.

  • Data strategy: New training resources combine public corpora with monolingual, aligned, synthetic, manually curated MeDLEy, and BOUQuET evaluation data.The strategy addresses gaps in long-tail languages, domains, registers, and evaluation coverage.
  • Synthetic and mined data: 270 million sentences span more than 2,000 languoids, with 51% from low-resource, 26% from very low-resource, and 23% from high-resource languoids.The resulting data are divided according to NLLB-based resource categories.
  • Synthetic and mined data: Synthetic data improves MT performance for lower-resource languages, while increasing backtranslation generally improves the other language-resource buckets.The paper also reports that filtering strategies affect outcomes across language groups.
  • Limitations: Synthetic-data coverage is limited by Common Crawl snapshots, expensive backtranslation and mining, and reliance on stratified sampling; MeDLEy coverage also omits some language-specific phenomena.External vendors for translation and quality assurance may increase inaccuracies in low-resource languages.
  • MeDLEy: MeDLEy covers 109 low-resource languages with multicentric, domain-diverse, grammatically diverse, and multiway-parallel seed data.Its design enumerates grammatical functions, creates multilingual source paragraphs, parallelizes them through pivots, and translates them into low-resource languages.
  • MeDLEy: MeDLEy matches or outperforms baseline seed datasets in token-controlled fine-tuning and improves into-English results, though adding it to existing datasets usually yields modest gains.The findings extend across metrics and support synthetic-data generation from monolingual low-resource-language data.
  • Evaluation data: BOUQuET is manually created across eight non-English languages and has expanded to 275 languages, while three benchmarks collectively cover over 1,600 unique languages.These resources target broader domains, registers, language families, scripts, and cultural contexts than earlier benchmarks.

5 Translation Modeling Overview

The modeling overview specializes LLMs for multilingual translation through decoder-only and encoder–decoder architectures, supported by tokenizer expansion for long-tail languages. Extended tokenization improves efficiency and translation quality, while controlled experiments quantify gains from the 256K vocabulary.

  • Architectures: The paper investigates decoder-only LLM fine-tuning and an encoder–decoder Transformer derived from an LLM for translation.These are the two principal architectural strategies explored for specializing general-purpose language models.
  • Tokenization: OMT-NLLB uses separate encoder and decoder tokenizers, while OMT-LLaMA extends the original LLaMA3 vocabulary; all three tokenizers contain 256K tokens.The OMT-NLLB input tokenizer covers over 1.5K languages, whereas its output tokenizer extends LLaMA3 for 200 languages.
  • Tokenization: The tokenizer vocabulary grows from 128K to 256K through continued BPE merging, alongside more language-friendly pre-tokenization rules.The changes target rare writing systems and heavily diacritic languages.
  • Tokenization: Reduced fertility increases training and inference throughput and usually improves translation by reducing the capacity spent reconstructing words from subwords.The paper notes that this relationship is typical but not universal.
  • Tokenizer ablation: 26% relative ChrF++ improvement occurs out-of-English (17.8 →22.5), while into-English improves 7% (35.9 →38.7) with the extended tokenizer.The ablation compares LLaMA3.2 1B Instruct models trained for 30K steps with identical data and hyperparameters.

6 Decoder-only Modeling

The decoder-only OMT-LLaMA pipeline combines continual pretraining, supervised fine-tuning, reinforcement learning, and retrieval augmentation to specialize LLaMA models for multilingual translation. SFT and RL consistently improve translation quality, while retrieval augmentation improves automatic metrics, especially with more examples.

  • Training pipeline: OMT-LLaMA uses continual pretraining on monolingual and parallel documents, followed by instruction and translation post-training.The post-training stages apply supervised fine-tuning and reinforcement learning.
  • Reinforcement learning: Diverse instruction data enables stable and generalizable RL gains, whereas narrowly templated data improves in-distribution performance but generalizes poorly.The final RL setup uses DAPO with larger groups, asymmetric clipping, dynamic sampling, KL regularization, and a balanced ChrF++/MetricX reward.
  • Supervised fine-tuning: SFT improves translation quality across BOUQuET and FLoRes+ evaluations, with gains especially visible for English-to-other directions and harder subsets.On full FLoRes+, English-to-other performance rises from 0.609 to 0.656; on FLoRes+-hard →en, it rises from 0.376 to 0.382.
  • Reinforcement learning: RL adds smaller but consistent improvements over SFT across BOUQuET and FLoRes+, without degrading any evaluated direction.On FLoRes+-hard, →en improves from 0.382 to 0.393.
  • Retrieval augmentation: Retrieval augmentation consistently improves all automatic metrics, with larger gains when more examples are available.Sentence-level gains are stronger than paragraph-level gains, particularly for smaller models, because the retrieval database is dominated by sentence- or word-level examples.

7 Encoder-Decoder Modeling

OMT-NLLB is a compact encoder–decoder system that combines cross-lingual encoder alignment with monolingual autoencoding and token-level attention. Its staged training removes the pooled-representation bottleneck and yields the strongest results after end-to-end fine-tuning.

  • Model overview: OMT-NLLB is a 3B-parameter Transformer translating from 1,600 source languages into 250 target languages.The model uses an encoder–decoder architecture based on the NLLB design.
  • Stage 1: aligned encoder and autoencoding: The first training stage combines parallel translation data with monolingual autoencoding through a frozen cross-lingually aligned encoder.Autoencoding increases training data, particularly for low-resource languages, and exposes the decoder to greater linguistic variation.
  • Representation bottleneck: The original pooled sentence representation creates a bottleneck by restricting fine-grained information passed from encoder to decoder.This prevents full use of token-level cross-attention as in standard Transformer MT.
  • Staged optimization: Decoder warm-up replaces pooled representations with token-level cross-attention before end-to-end fine-tuning.Only decoder parameters are updated during warm-up, then encoder and decoder are jointly optimized on parallel data.
  • Stage 1 results: 105 of 114 languages receiving added autoencoding data improve in the language-specific analysis.The comparison evaluates Decoder Enhancing MT+AE against Decoder Enhancing MT.
  • Final results: The final OMT-NLLB model achieves the strongest results while retaining the compact 3B-parameter size.End-to-end fine-tuning lets the encoder adapt representations to translation and fully optimize token-level interactions.

8 Proposed Evaluation Metrics and Dataset

This section develops an evaluation framework for omnilingual MT, combining scalable human protocols, multilingual metric benchmarking, and new evaluation datasets. XSTS+R+P adds register and paragraph-level considerations to XSTS, while Met-BOUQuET enables broad metric reliability assessment.

  • Motivation: Human evaluation is relevant for MT comparison but costly, slow, difficult to scale, and dependent on protocol quality.The protocol must remain compatible with many languages while capturing register and contextual ambiguity.
  • XSTS+R+P: XSTS+R+P extends XSTS with sentence-level register scoring and paragraph-level penalties for confusion despite sentence-level semantic equivalence.It preserves XSTS’s focus on semantic equivalence while adapting evaluation to BOUQuET’s register diversity and paragraph structure.
  • Protocol comparison: XSTS+R+P scores correlate positively with XSTS at 0.65 and RSQM at 0.62 using Kendall’s Tau.The protocol comparison also examines score distributions across XSTS+R+P, XSTS, and RSQM.
  • XSTS+R+P: 0.80 is the reported mean Krippendorff’s α for XSTS+R+P, substantially exceeding the baseline protocols and typical comparable translation-evaluation levels.Agreement is calculated with a squared-distance penalty.
  • Met-BOUQuET: Met-BOUQuET is a highly multilingual, multi-way parallel annotation dataset and benchmark for evaluating MT metrics and quality estimation.It includes human annotations and supports benchmarking metric reliability across diverse languages and language pairs.
  • Met-BOUQuET: Met-BOUQuET contains 73% non-English directions, comprising 118 directions without English.The dataset includes XSTS+R+P annotations and experimental XSTS and RSQM annotations.

8.3.3 Experimental Setup

The study evaluates multilingual quality estimation and toxicity detection across highly multilingual settings, using language-identification adjustments, cross-lingual embeddings, and controlled comparisons. Results emphasize strong multilingual performance alongside limits in automatic evaluation and representation quality for some languages.

  • BLASER 3 surpasses MetricX-24 by +0.08 and xCOMET-XL by +0.12 on multilingual quality estimation.
  • Automatic metrics often miss wrong target languages, paragraph-level context, and mistranslated keywords that substantially alter human judgments.
  • Reference-free BLASER 3 with LID adjustment, plus source-and-reference MetricX and xCOMET with LID adjustment, are recommended for highly multilingual translation evaluation.
  • OmniTOX achieves an overall ROC AUC of 0.845, outperforming MuTox by +0.058 points.
  • The embedding upgrade contributes +0.052 ROC AUC, while classifier optimization adds +0.006, indicating representation quality is the primary improvement source.
  • Baseline ZS reaches 0.793 overall ROC AUC and a 0.821 mean per-language ROC AUC when trained only on English and Spanish.

9 MT Results

OMT models substantially broaden multilingual translation quality and coverage, especially for lower-resourced languages and long-tail generation. Results also show architecture- and scale-dependent trade-offs, while automatic evaluation remains limited for long-tail outputs.

  • OMT systems outperform strong baselines by a significant margin on mid- and low-resourced translation directions in FLoRes+ evaluations.
  • Long-tail understanding and generation: 1,045 languages are cases where OMT-LLaMA 8B strictly outperforms all baselines on Bible XX-En evaluation.
  • Long-tail understanding and generation: OMT models maintain meaningful quality across about 1,200 languages, whereas baseline quality becomes near-random at about 300–400 languages.
  • Evaluation limitations: Automatic metrics cannot adequately evaluate grammaticality, fluency, or naturalness for long-tail translations, limiting certainty about how many outputs are good enough.
  • Model size and architecture: Across all size categories, OMT models outperform baseline models of corresponding size on BOUQuET translation-quality evaluations.
  • Model size and architecture: OMT-LLaMA can generate over 1,000 languages, while OMT-NLLB generates 250; OMT-NLLB often has higher quality for high- and mid-resourced outputs.
  • Human evaluation: Manual evaluation finds OMT-LLaMA better than baselines in 44 of 57 directions (77%), with average quality rising from 2.67 to 3.38.The reported increase is described as a shift from “useless” to “useful” for many directions, but remains below scores of 4 and 5.

10 Extensibility of OMT-LLaMA

The paper extends OMT-LLaMA for selected difficult languages using focused fine-tuning and retrieval-augmented translation. These techniques improve targeted translation in complementary ways, although fine-tuning can harm into-English performance.

  • Experimental setup: Focused adaptation targets 25 difficult languages selected largely from low- and very-low-resource BOUQuET languages, plus five mid-resourced languages.
  • Data: The extension experiments use language-specific parallel data ranging from 300K examples for Swahili to 11K for Ngambay.
  • Fine-tuning: Fine-tuning improves out-of-English translation on average but is detrimental to into-English translation in the base experiment.
  • Retrieval augmentation: Retrieval-augmented translation almost always improves both baseline and fine-tuned models.
  • Comparison with LLaMA-base: Combining instruction fine-tuning and retrieval does not make base LLaMA outperform unadapted OMT-LLaMA.
  • Conclusions: Fine-tuning is particularly effective for translation into challenging languages, while retrieval augmentation is essential for improving translation from those languages.

11 Conclusion

Omnilingual MT scales multilingual translation beyond 1,000 languages by combining broad data creation, specialized architectures, and expanded evaluation. Its results show stronger coverage, efficient specialization, improved generation, and publicly available resources for continued research.

  • Conclusion: More than 1,000 languages required coordinated changes to data, modeling, and evaluation, including MeDLEy and BOUQuET resources.OMT uses massive public corpora, targeted data creation, extended tokenizer vocabularies, two architectures, and a three-stage OMT-NLLB training strategy.
  • Conclusion: 1B to 8B specialized MT models match or exceed the MT performance of a 70B LLM, yielding a clear efficiency–performance Pareto advantage.The reported efficiency supports strong translation quality in low-compute real-world settings.
  • Conclusion: OMT improves cross-lingual transfer and expands coherent generation, while finetuning and retrieval-augmented generation provide further language- and domain-specific gains.These targeted techniques help when additional data or domain knowledge is available.
  • Conclusion: BOUQuET, Met-BOUQuET, and the adjacent leaderboard are freely available, enabling reproducible evaluation and further multilingual data development.The authors also provide MeDLEy guidelines and model, evaluation, and toxicity-analysis recipes.

12 Contribution Statements

The contribution statements cover the team’s leadership, modeling, data, evaluation, linguistic, and program-management roles, alongside the construction of MeDLEy. MeDLEy is designed for broad grammatical and domain diversity through structured creation, pivot translation, and professional translation into low-resource languages.

  • Contribution Statements: The team assigned distinct roles spanning MeDLEy, OMT-NLLB, OMT-LLaMA, data mining, retrieval-augmented translation, evaluation, annotation, and partnerships.Named contributions include continual pretraining, vocabulary extension, BOUQuET and Met-BOUQuET, BLASER 3, and OmniTOX.
  • MeDLEy: MeDLEy aims to provide domain-diverse and grammatically diverse bitext that language communities can extend through simple translation.Its framework is intended to avoid dependence on target-language-specific linguistic expertise.
  • MeDLEy: Five domains, five source languages, linguistic templates, and expert-created paragraphs structure the source data for accessibility and grammatical diversity.The domains are informative, dialogue, casual, narrative, and instruction-response; source languages are English, Mandarin, Russian, Spanish, and German.
  • MeDLEy: Eight pivot languages and 109 low-resource target languages were selected before professional translations produced grammatical-diverse target-language bitext.Selection considered global L2 coverage, translator availability, previous open-source coverage, and language-family representativeness.
  • MeDLEy: MeDLEy covers 18 grammatical categories and 61 features, with two features dropped in Batch 2 because they lacked generalized transfer in translation.The features represent common cross-linguistic grammatical phenomena.
  • Sentence alignment: Sentence alignment combines SaT boundary probabilities with bidirectional semantic similarity, selecting the best-scoring candidate in each window.SONAR embeddings compare source text with left and right target segments to produce 1:1 aligned sentence segments.

A.8.1 Feature distribution analysis

The feature-distribution analysis compares grammatical diversity across datasets using entropy-based measurements. MeDLEy generally provides the most balanced distributions and includes rarer grammatical features in both English and Hindi analyses.

  • Feature distribution analysis: Higher entropy indicates a more balanced distribution over a grammatical feature paradigm.The analysis compares feature distributions across datasets using Stanza annotations and heuristics.
  • Feature distribution analysis: MeDLEy shows the highest entropy in 5 of 9 English feature categories and often contains rarer features such as first person, perfect aspect, and clausal negation.NLLB-Seed instead concentrates on Wikipedia-associated features including past tense, indicative mood, and third person.
  • Feature distribution analysis: In Hindi, MeDLEy often has the highest distribution entropy and more balanced tense, verbal-valency, and negation-type distributions.The Hindi comparison uses automatically translated SmolSent and NLLB-Seed, so translation may affect the findings.

A.8.2 Feature retention study

The feature-retention study examines whether grammatical functions survive one- and two-hop translation across Russian, Spanish, and English. Most morphosyntactic features transfer above 50%, but features can be lost, redistributed, or resurface in later hops depending on the languages involved.

  • Feature retention study: The analysis is limited to languages in the dataset intersection for which parsers and linguistic expertise are available, with English as the only such language.This constrains the scope of the parser-supported analysis.
  • Feature retention study: Most morphosyntactic features have transfer rates above 50%, with marked word order as the main exception.The study evaluates 10 features across Spanish- and Russian-origin paragraphs and one- or two-hop translations.
  • Feature retention study: The imperative is the most robust feature, copied in both one- and two-hop translations, while verbal-morphology features generally have the highest retention.These include past tense, passive voice, and imperative mood.
  • Feature retention study: Marked word order can be lost through English because English has more rigid word order and fewer marked-word-order patterns than Russian and Spanish.The study reports that such forms can still be cued in a later target translation.
  • Feature retention study: Features lost in one translation hop can reappear in a subsequent hop, including marked word order in Spanish and Russian two-hop translations.The qualitative examples also show redistribution into other rare features such as instrumental case.

A.10 More details on the experimental setup

The experiments compare NLLB and LLaMA translation paradigms, using supervised fine-tuning on language-specific directions and seed bitext datasets. Training and inference use distinct efficiency-oriented configurations for each architecture.

  • Models and paradigms: NLLB-200-3.3B represents sequence-to-sequence MT, while LLaMA-3.1-8B-Instruct represents decoder-only, LLM-based MT.NLLB uses language tags and beam search; LLaMA uses a minimal source-to-target translation prompt.
  • Training procedure: Supervised fine-tuning maximizes target-sequence log-likelihood given source text with teacher forcing, selecting checkpoints by held-out validation performance.Models are fine-tuned and evaluated separately for each language pair and direction.
  • Data and languages: Five evaluation languages—Bambara, Mossi, Wolof, Yoruba, and Ganda—are covered by the baseline datasets, MeDLEy-109, and the evaluation datasets.SmolDoc samples are split into sentence-level chunks using supplied sentence alignments.
  • Input formatting: NLLB examples encode source and target language codes alongside source and target text, whereas LLaMA uses an instruction prompt specifying the translation direction.The NLLB representation uses reserved tokenizer tokens for supported language codes.
  • Implementation: Fine-tuning uses A100 GPUs, mixed precision, dynamic padding for NLLB, and packing, FlashAttention-2, and FSDP for LLaMA.Inference uses ctranslate2 for NLLB and vLLM for LLaMA; hyperparameters are reported in Tables A.9 and A.10.

A.11 More details on the experiment results

The experiments find that seed data improves translation, while MeDLEy generally matches or exceeds baseline datasets but produces only modest gains when added at small scale. Results remain especially weak for English-to-low-resource directions, and NLLB retains a strong low-resource advantage over LLM-based MT.

  • Seed-data effects: Seed data improves performance over the no-seed baseline, especially for LLaMA.The comparison evaluates controlled data mixtures with matched training conditions.
  • Model comparison: NLLB achieves higher baseline and fine-tuned performance across the compared low-resource languages than LLM-based MT.The result is consistent with the reported advantage of smaller sequence-to-sequence models in low-resource MT.
  • Dataset comparison: MeDLEy matches or outperforms baseline datasets under token-controlled evaluation and shows gains in the into-English direction.The trend is also observed with xCOMET and MetricX.
  • Dataset comparison: 54% of NLLB-Seed examples are difficult-to-translate texts, compared with 10.41% in the comparison data.The passage attributes this difference potentially to technical or obscure terminology that may hinder community translators.
  • Dataset combination: Adding MeDLEy to existing seed datasets yields generally small improvements, indicating challenges for manual data collection at a few thousand sentences.MeDLEy nevertheless includes 92 languages not covered by SMOL.
  • Overall performance: Translation scores remain low in general regardless of seed dataset, especially in the en-xx direction.The broader evaluation reports chrF++, xCOMET, and MetricX as complementary metrics.

B.3 Comparison to other MT metrics evaluation datasets

Met-BOUQuET is designed for multilingual, parallel evaluation across languages and directions, extending existing datasets with broad coverage and detailed metadata. Its structure enables comparisons beyond datasets restricted to a single source language.

  • Dataset comparison: Table B.1 compares MT-metric evaluation datasets by protocol, language coverage, language pairs, source sentences, systems, and domains.The comparison includes MLQE, IndicMT Eval, AmericasNLP Task 3, and NLLB.
  • Dataset comparison: Existing datasets usually use different source sentences across directions, while AmericasNLP Task 3 and IndicMT Eval preserve parallelism within one source language.Their multilingual scope remains limited because that parallelism does not extend across multiple source languages.
  • Met-BOUQuET design: Met-BOUQuET assigns unique identifiers to source paragraphs and sentences regardless of source language, enabling cross-lingual comparisons across directions.This design combines parallel evaluation with broader multilingual benchmarking.
  • Coverage: Met-BOUQuET Round 1 contains 100% bidirectional pairs and 60% non-English directions, while Rounds 1 and 2 together include 118 non-English directions.NLLB is reported as the next closest dataset with 20 non-English directions.
  • Metadata: Met-BOUQuET provides domain, register, linguistic, and producing-system annotations, enabling granular performance benchmarking despite containing a single target output.These metadata are inherited from BOUQuET.
  • Score distributions: English-involving pairs tend to receive higher XSTS+R+P consensus scores than non-English pairs, which contain more score-1 translations.Figure B.1 reports these distributions and averages for Met-BOUQuET Round 1.
Loading 2603.16309v3…