Source-linked AI summary
WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects
Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, Markus Freitag
TL;DR
WMT datasets typically cover only around ten language pairs, motivating broader benchmarks for multilingual LLM machine translation. WMT24++ extends WMT24 to 55 languages and dialects and evaluates MT providers and LLMs with automatic metrics. Frontier LLMs outperform standard MT providers across all 55 languages by these metrics, but the conclusions require human-based verification.
Problem
WMT datasets typically cover around ten language pairs, creating a need for MT benchmarks spanning more languages as LLM multilingual capabilities grow.
Method
WMT24++ extends the WMT24 dataset to 55 languages and dialects and benchmarks MT providers and LLMs using automatic evaluation metrics.
Results
Frontier LLMs outperform standard MT providers in all 55 languages according to automatic metrics, which also score their translations above human-written references and post-edits.
Takeaways & Limitations
WMT24++ contributes references and post-edits intended to support research on multilingual LLM machine translation across a large language set.
Takeaways & Limitations
Because automatic evaluation metrics have limitations, the paper’s conclusions should be verified through human-based evaluation.
Abstract
from arXiv · showhide
As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages and dialects in addition to post-edits of the references in 8 out of 9 languages in the original WMT24 dataset. The dataset covers four domains: literary, news, social, and speech. We benchmark a variety of MT providers and LLMs on the collected dataset using automatic metrics and find that LLMs are the best-performing MT systems in all 55 languages. These results should be confirmed using a human-based evaluation, which we leave for future work.
1 Introduction
WMT24++ expands WMT24 to 55 languages and dialects with new human references and post-edits, then benchmarks MT providers and LLMs using automatic evaluations. Frontier LLMs outperform standard MT providers across all 55 languages according to those metrics, but the authors caution that human evaluation is still needed.
- 55 languages and dialects are covered by the extended WMT24++ benchmark.
- 46 new languages receive human-written references and subsequent post-edits.
- Eight of the nine original WMT24 languages receive new post-edit corrections for their references.
- Automatic evaluations benchmark a variety of MT service providers and LLMs across the covered languages.
- Frontier LLMs outperform standard MT providers in all 55 languages according to automatic metrics.
- Automatic-metric limitations, including possible bias against human translations, make large-scale human evaluation necessary before definitive MT-quality claims.
2 Dataset Collection
The dataset reuses WMT24's 998 English paragraph sources across four domains and extends them with professional translations, post-edits, machine translations, and automatic evaluations for 55 language pairs.
- Reference & Post-Edit Collection: WMT24++ reuses 998 English paragraph sources from literary, news, social, and speech domains.
- Reference & Post-Edit Collection: References and subsequent post-edits are collected for 46 new languages and dialects, while eight original language pairs receive post-edits.
- Reference & Post-Edit Collection: Professional translators receive document context and source URLs to support paragraph-level translation and post-editing.
- Reference & Post-Edit Collection: Post-editing prioritizes high-quality translations over restricting edits to the original references, so some post-edits are complete rewrites.
- Reference & Post-Edit Collection: Native speakers verify that sampled post-edits represent the intended regional dialects.
- Machine Translation Collection: Machine translations for all 55 languages and dialects come from MT providers and LLMs, using Intento for traditional providers and zero-shot regional prompting for LLMs.
- Machine Translation Collection: The benchmark includes Aya 23, Claude 3.5 Sonnet, Command R+, DeepL, Gemini, Google Translate, GPT-4o, OpenAI o1, OpenAI o1-mini, Microsoft Translate, Tower-70B, and Yandex Translate.
- Automatic Evaluation Metrics: Translations are scored with reference-based and reference-free automatic metrics, including MetricX-24 and MetricX-24-QE, with post-edits serving as references for reference-based metrics.
3 Analysis
MetricX-24-QE generally scores post-edits at least as highly as references and frontier LLM translations at least as highly as human translations across the 55-language benchmark. However, cross-language metric variation and known bias against human translations prevent definitive quality conclusions.
- Human and machine translation quality: Except for ar_EG and ar_SA, post-edits receive equal or better MetricX-24-QE scores than the original references.The authors hypothesize that the Arabic difference reflects references written in Modern Standard Arabic rather than regional variants.
- Human and machine translation quality: The best MT system is roughly equal or better than human translations for every language, often by a large margin.The authors caution that QE metrics systematically rate human translations lower than human evaluators do.
- Comparing across languages: MetricX scores vary significantly across languages, from -2 to -6, indicating a large cross-language gap.Human and machine translation scores follow nearly identical trends, leading the authors to hypothesize that the metric behaves differently across languages.
- System evaluation: MetricX-24 rankings place OpenAI o1, Claude, and Gemini highest for every language, with average ranks of 1.5, 1.9, and 2.1.Their absolute metric scores are also reported as very similar.
- System evaluation: Frontier LLMs rank above traditional MT providers according to automatic metrics, but the comparison may trade quality against speed and cost.The authors speculate that provider models are significantly smaller and faster, while noting that exact system details are unknown.
- Evaluation limitations: Human-based evaluation is needed because automatic metrics are biased against human translations and largely untested in most of the 55 languages.The authors state that they intend to perform this evaluation in future work.
4 Related Work
The work builds on WMT’s recurring MT datasets while addressing their typically limited language coverage. It complements broad multilingual benchmarks such as FLORES with paragraph-level translations spanning four domains.
- WMT datasets: WMT shared tasks have collected datasets and benchmarked state-of-the-art MT systems annually since 2006.This work directly builds on the WMT24 dataset and expands its language coverage.
- WMT datasets: WMT datasets typically include around ten language pairs, motivating broader coverage as LLM multilingual capabilities grow.The authors identify large-language benchmark collection as critical for multilingual LLM research.
- Multilingual MT benchmarks: FLORES covers 200 languages with an emphasis on low-resource languages and uses Wikimedia sentences from three domains.The listed domains are news, children’s books, and travel.
- Multilingual MT benchmarks: AfriCOMET and IndicMT extend or build on FLORES with additional translations or human annotations for African and Indian languages.These efforts target regional language coverage within the broader multilingual benchmark ecosystem.
- This work: This work complements FLORES by providing an updated dataset with paragraph-level translations across four domains: news, literary, speech, and social.The supplied passage contrasts its coverage with FLORES’s sentence-level, three-domain source texts.
5 Conclusion
WMT24++ extends WMT24 to 55 languages and dialects and evaluates MT systems across this expanded benchmark. The experiments find strong performance from frontier LLMs, but both conclusions require human-based verification because automatic metrics are limited.
- The experiments find that MT systems appear to generate higher-quality translations than humans and that frontier LLMs are highly capable across many languages.
- Automatic-metric limitations prevent definitive conclusions, so the reported findings should be verified through human-based evaluation.
Limitations
The dataset collection cannot guarantee that every reference translation and post-edit is of the highest quality across all 55 languages. The authors also cannot draw definitive conclusions from automatic evaluation alone and leave in-depth human evaluation for future work.
- Every reference translation and post-edit cannot be guaranteed to have the highest quality across all 55 languages.The authors nevertheless performed quality checks, including QE-based analysis and human spot-checks, to make the data as high-quality as possible.
- Automatic evaluation limitations prevent definitive conclusions about some claims, while in-depth human evaluation is out of scope for this work.
A Prompts
The appendix documents the prompts used to collect translations from LLMs and to evaluate translation quality.
- Figure 3 shows the prompts used to collect translations from LLMs, with two prompts unintentionally used across different systems.
- Figure 4 shows the prompts used to evaluate translation quality with Gemini-1.5 Pro.
B Additional Results
The additional results compare edit distances, QE-based translation quality, and metric-based system rankings. Rankings and scores exclude 38 bad source texts, and cross-language absolute metric values are not directly comparable.
- Some languages received clearly more editing than others, as shown by the reference-to-post-edit edit-distance distributions.
- Figure 6 compares reference, post-edit, and best-MT quality using various QE metrics, whose language orderings generally do not show extreme disagreement.The passage attributes this agreement likely to correlations among the metrics.
- Rankings and scores are calculated without 38 bad source texts, and absolute metric values are not comparable across languages.Systems supporting only languages with high metric scores may therefore be favored.
- Table 3 indexes the figures containing each metric’s system scores and rankings.
C Failure to Translate Details
The evaluation removed 38 low-quality source texts that caused most translation failures, then retained remaining refusals and tested their effect on results.
- 38 low-quality source texts caused the vast majority of translation failures and were removed from subsequent analyses.These sources added noise and were mostly low-quality content.
- After removal, some LLMs still failed to produce translations, with GPT-4o1 refusing gu_IN inputs largely because they contained profanities.The refusal pattern differed across languages despite identical sources.
- The evaluation retained additional refusals because an all-systems-translated-segments analysis produced results not significantly different from the main evaluation.
D Metric Bias Against Human Translations
The paper cautions that automatic quality metrics may systematically undervalue human translations, while the LLM evaluation also used unintentionally different prompts across systems.
- QE metrics ranked human translations lower than human evaluators did in every tested WMT24 Metrics Shared Task setting.The comparison used MQM as the human-based ground-truth ranking alongside MetricX-24-QE and XCOMET-QE.
- Two different prompts were unintentionally used to generate translations from different LLM groups.Gemini-1.5 Pro, Claude-3.5, and GPT-family models used one prompt, while Unbabel-Tower70B, CommandR-plus, and Aya23 used another.
- The authors caution that higher automatic scores for MT systems cannot establish that they are higher quality than human translations.The metrics may unfairly penalize human translations.
E Screenshot Examples
The paper documents screenshot-based source examples from speech and social domains and presents figures comparing translation quality, edit distances, and system rankings across metrics and languages.
- 94% of original sources were available as full-page, variable-length screenshots, covering 85% of total segments.Screenshots had a uniform width of 750px and were preserved alongside original URLs.
- Speech-domain screenshots include YouTube thumbnails and full video descriptions, while social-domain screenshots include Mastodon thread context, embedded images, previews, and replies.These sources can provide illustrative or visually situated language for multimodal approaches.
- Figures 6 through 9 compare references, post-edits, and best MT outputs and show system rankings using significance clusters across metrics.Within a cluster, no system is significantly better than all others; worse clusters are statistically separated.
- Figure 19 counts outputs containing text other than a translation after 38 bad source segments were removed.For gu_IN, OpenAI o1 refused profanity-containing inputs, unlike its refusal rate in other languages.