Source-linked AI summary
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation
Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, Arul Menezes
TL;DR
Automatic metrics are widely used to choose between MT systems, but evidence about their reliability has been limited by small human-judgement sets and mismatched evaluation scenarios. The paper evaluates pairwise ranking accuracy across a large multilingual collection, finding COMET strongest overall and presenting evidence that overreliance on BLEU can hinder MT development. It releases 2.3 M human judgements covering 4380 systems for replication and further analysis.
Problem
Automatic metrics often determine which MT system is considered better, while prior human-correlation evidence has been limited in size and often does not reflect pairwise comparisons.
Method
The study introduces pairwise system-level metric evaluation using human judgements as gold rankings across language pairs, domains, and system comparisons.
Results
COMET is the best-performing metric, COMET-src is highly accurate despite being reference-free, and the authors present indirect evidence that BLEU overuse negatively affects MT development.
Takeaways & Limitations
The authors recommend COMET as the primary metric, ChrF as a secondary or unsupported-language metric, paired significance testing, and public system outputs.
Takeaways & Limitations
Statistical significance tests rely on independence and distribution assumptions that are rarely true and do not account for random seed variation, requiring caution with small metric improvements.
Abstract
from arXiv · showhide
Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic metric guides research directions and industrial developments by deciding which models are deemed better. Evaluating metrics correlations with sets of human judgements has been limited by the size of these sets. In this paper, we corroborate how reliable metrics are in contrast to human judgements on -- to the best of our knowledge -- the largest collection of judgements reported in the literature. Arguably, pairwise rankings of two systems are the most common evaluation tasks in research or deployment scenarios. Taking human judgement as a gold standard, we investigate which metrics have the highest accuracy in predicting translation quality rankings for such system pairs. Furthermore, we evaluate the performance of various metrics across different language pairs and domains. Lastly, we show that the sole use of BLEU impeded the development of improved models leading to bad deployment decisions. We release the collection of 2.3M sentence-level human judgements for 4380 systems for further analysis and replication of our work.
1 Introduction
Automatic metrics are widely used to compare MT systems, despite evidence that they can poorly approximate human judgement and mislead development. This study targets pairwise system ranking and proposes best practices for more reliable metric-based decisions.
- Automatic metrics often replace expensive, time-consuming human evaluation when comparing two MT systems.
- Prior metric studies commonly analyze sentence-level correlations or isolated system-level correlations, which do not match pairwise system comparisons used in practice.
- The study designs a pairwise system-level evaluation methodology and applies it to the largest human-judgement collection reported by the authors.
- The authors also examine metric reliability across language pairs, text domains, and statistical tests over automatic scores.
- Recommended practice is to use COMET as the primary metric, ChrF for unsupported languages or as a secondary metric, paired significance tests, and public system outputs.
2 Data
The study assembles a large, multilingual evaluation resource from professionally assessed MT systems and publishes supporting materials where legally possible. Its comparisons use controlled campaigns and shared sentences, while the analyzed systems are all neural.
- The released collection contains human judgements, metadata, metric scores, and code, but proprietary test sets and system outputs cannot be released for legal reasons.
- 101 languages across 232 translation directions are evaluated, with test sets averaging 1017 sentences and an English-centric distribution.
- Human annotations from mid-2018 through early 2021 use identical settings and the same annotator pool to maintain similar distributions and characteristics.
- Each campaign compares two to four systems on around 500 shared test sentences, with each system pair assessed by five annotators.
- Bilingual annotators rate translations from 0 to 100 against source sentences without references using source-based Direct Assessment.
- The analysis covers 4380 systems and 2.3 M annotated sentences, and uses paired comparisons of systems evaluated under equal conditions.
- System pairs represent model improvements, state-of-the-art evaluations, or comparisons with third-party systems; all analyzed models use neural architectures.
3 Automatic metrics
The evaluation compares string-based metrics with pretrained-model metrics, emphasizing language-agnostic methods and implementation comparability. Metric properties include reference dependence, training data, language support, and evaluation granularity.
- The study evaluates promising recent metrics and widely used language-agnostic metrics, excluding methods that support only small language sets.
- String-based metrics compare substrings between references and outputs, making their scores largely dependent on reference quality but relatively easy to diagnose.
- Pretrained metrics use neural models with the source, reference, or both, allowing evaluation less strictly dependent on reference wording but influenced by training data.
- Metrics are evaluated with recommended implementations, and negatively oriented error metrics such as TER have their scores negated.
- Comparisons use language pairs supported by all evaluated metrics to ensure comparability.
4 Evaluation
The paper evaluates metrics through pairwise score differences rather than isolated system correlations. Because metric scales vary across language pairs, it uses binary ranking accuracy and supplements it with significance testing and bootstrap confidence intervals.
- 4.1 Pairwise score differences: Previous work mainly correlated individual system scores with human judgements, whereas this study treats pairwise comparison as the common practical evaluation scenario.
- 4.1 Pairwise score differences: Each system pair is drawn within a campaign, where systems are evaluated under equal conditions and produce one, three, or six pairs for two, three, or four systems.
- 4.1 Pairwise score differences: All metrics correlate positively with human judgements, but COMET has the smallest deviation and highest correlation; several metrics show inconsistent value ranges across language directions.
- 4.1 Pairwise score differences: Because metric scales are not equal across language pairs, Pearson’s and Spearman’s correlations cannot be used for pairwise evaluation.
- 4.2 Pairwise system-level metric quality: Pairwise accuracy measures whether metric and human score differences have the same sign, treating human rankings as gold labels.
- 4.2 Pairwise system-level metric quality: Accuracy can be overestimated when small human differences happen to share a sign with metric differences, so the analysis also excludes statistically indistinguishable pairs.
- 4.2 Pairwise system-level metric quality: Bootstrap confidence intervals define comparable best-performing metrics as those within the best metric’s 95% interval.
5 Results
Across language, domain, and system subsets, pretrained metrics generally rank MT system pairs more accurately than string-based metrics, with COMET typically leading. Statistical testing improves reliability, while BLEU’s role in incremental model development is associated with rejecting models preferred by humans.
- 5.1 Which metric is best suited for pairwise comparison?: COMET reaches the highest accuracy for pairwise system ranking, while COMET-src is the runner-up despite requiring no human reference.Prism-src performs poorly mainly for into-English directions.
- 5.1 Which metric is best suited for pairwise comparison?: Pretrained metrics generally outperform string-based metrics, and ChrF is the strongest string-based alternative to BLEU.The ordering remains similar across analyzed subsets, supporting the general validity of the results.
- 5.1 Which metric is best suited for pairwise comparison?: Removing statistically indistinguishable system pairs increases accuracy, but no metric reaches 100% accuracy even among strongly different systems at alpha=0.001.Metric ordering remains unchanged after removing comparable pairs.
- 5.2 Are metrics reliable for non-English languages and other scenarios?: Pretrained metric rankings remain broadly stable across language directions and domains, although non-Latin target scripts slightly favor ChrF and discussion data reduce string-based metric accuracy.These findings suggest COMET is not overfitted to WMT news or WMT languages.
- 5.3 Are statistical tests on automatic metric worth it?: Applying paired bootstrap resampling at alpha=0.05 increases accuracy by around 10% for all metrics, but 17.3% of COMET’s non-significant pairs are judged significantly different by humans.The authors therefore encourage statistical significance testing while recognizing type II errors.
- 5.4 Does BLEU sabotage progress in MT?: BLEU wins on incremental system pairs even though pretrained metrics lead on independent pairs, indicating that BLEU influenced development and caused rejection of human-preferred models.The authors characterize this as indirect evidence requiring evaluation of rejected systems with other metrics and human judgement.
6 Meta Analysis
The meta-analysis aggregates prior system-level metric correlations across language pairs to contextualize the paper’s findings. It confirms that pretrained methods outperform string-based metrics and that BLEU has not exceeded ChrF in any analyzed year, while one anomalous English-Chinese system affected earlier results.
- 6 Meta Analysis: The Hunter-Schmidt method combines prior correlations of different sizes across language pairs to generalize system-level findings.The analysis focuses on correlations with human judgement across languages rather than separately by translation direction.
- 6 Meta Analysis: The analysis selects metrics evaluated in multiple studies or previously promising, using settings close to recent recommendations or later evaluation studies.This defines the subset of metric variants included in the meta-analysis.
- 6 Meta Analysis: Pretrained methods outperform string-based methods, and BLEU has not had higher correlation than ChrF in any analyzed year.ESIM performs best overall among pretrained methods, while COMET leads after removing the suspicious system.
- 6 Meta Analysis: A submitted COMET score failed on one English-Chinese system with tokenized output, but replication produced a valid score and the system strongly skewed Pearson’s correlation.The authors therefore also report analysis excluding English-Chinese systems.
7 Discussion
The discussion favors pretrained metrics, especially COMET, while recommending ChrF as a secondary metric and cautioning against BLEU overuse. It also highlights statistical-testing and reproducibility practices, alongside important limits on the analysis and human-judgement assumption.
- Metric choice: Pretrained metrics generally achieve higher accuracy than string-based methods, although pretrained approaches are relatively new.The authors note that newer methods may have undiscovered drawbacks, including training-data biases, domain failures, or fluency-over-adequacy preferences.
- Metric choice: The authors recommend COMET as the primary metric and ChrF as a secondary metric or option for unsupported languages.They discuss metric preselection as a way to expose improvements missed by other metrics, but warn that it could encourage metric-hacking.
- Metric choice: COMET-src, a reference-free metric, shows high accuracy and may support automatic evaluation on monolingual domain-specific test sets.This extends metric use beyond settings requiring reference translations.
- Metric choice: BLEU’s overuse may negatively affect MT development, supporting its deprecation as the evaluation standard.The discussion cites prior evidence that 98.8% of MT papers use BLEU and presents indirect evidence linking its overuse to development problems.
- Evaluation practice: Paired statistical significance tests can increase the reliability of metric decisions, but small improvements require caution because common assumptions may fail and training seeds vary.The tests do not account for random seed variation across training runs.
- Evaluation practice: Publishing translated test-set outputs enables recalculation with different metrics and easier comparison across research.The discussion presents this as preferable to permanently relying on copied or obsolete metric scores.
- Limitations: The analysis cannot establish metric behavior for language pairs without English or for low-resource languages because the data contain few non-English systems and mostly high-resource pairs.Many directions also run from translationese into authentic text, an easier direction for systems to score highly by human judgement.
- Limitations: The study treats human judgement as the gold standard, although human methods and annotators may fail to capture true assessment.The discussion notes that direct assessment cannot explicitly mark critical errors in DA, which are instead reflected through low scores.
8 Conclusion
The study finds that pretrained metrics, especially COMET, are more reliable than string-based metrics for pairwise system ranking, while statistical testing improves decision reliability. It also cautions against Pearson’s correlation across languages and recommends publishing outputs for future score recalculation.
- Metrics can use different scales across languages, so Pearson’s correlation cannot be used for pairwise metric evaluation.
- COMET is the best-performing metric, while ChrF is the strongest string-based method.
- The study presents indirect evidence that overusing BLEU negatively affects machine translation development.
- Statistical testing largely increases the reliability of pairwise decisions based on automatic metric scores.
- The authors recommend publishing translated research-system outputs to enable future comparisons and score recalculation.
A Metrics Implementation Details
The evaluation uses standard or recommended implementations and parameters for the compared metrics, with specified model versions and tokenization choices. BLEURT receives additional treatment because its recommended model is fine-tuned for English only.
- The study uses default or recommended parameters to simulate standard metric usage.
- BLEU, ChrF, and TER use SacreBLEU version 1.5.0 with the mteval-v13a tokenizer, except for Chinese and Japanese.
- BLEURT uses bleurt-base-128, is fine-tuned for English only, and includes bleurt-large-512 because it performed better.
- COMET uses the recommended wmt-large-da-estimator-1719 model, while COMET-src uses wmt-large-qeestimator-1719.
- Prism and Prism-src, ESIM, CharacTER, EED, and BERTScore are evaluated using specified public implementations and revisions.
B Confidence Interval for Metric Accuracy
The confidence interval procedure assesses whether the best metric’s accuracy is statistically distinguishable from competitors using paired bootstrap resampling. It repeatedly resamples system pairs and compares metric accuracies.
- The bootstrap method generates multiple resamples with replacement and calculates accuracy on each resample to estimate confidence intervals.
- 10 000 resamples are created by drawing system pairs with replacement from the full collection.
- Each resample produces an accuracy value for every metric, which is compared with the best-performing metric α.
- A competing metric is considered on par with α at 95% significance when α outperforms it less than 95% of the time.
C Comparing Statistical Tests
The comparison of statistical tests distinguishes significant from non-significant system differences and examines how metric decisions align with human judgements. It also documents evaluation scope, figure interpretation, and a possible post-edited-data issue in Polish–English results.
- Human-j judgement equivalence is assessed with Wilcoxon’s test at alpha 0.05, while automatic metrics use paired bootstrap resampling.
- When both tests identify significant differences, metric accuracy is evaluated over pairs judged truly different or classified as type I errors.
- The second scenario measures metric non-significance when human judgements identify different systems, while acknowledging that test powers and type II errors differ.
- Figure 2 plots metric-score differences against human-judgement differences, with colors distinguishing translation directions and pink quadrants marking ranking disagreements.
- The evaluation covers 232 translation directions and reports language-pair accuracy for pairs with at least 20 system comparisons.
- Polish–English results may reflect a test set consisting of post-edited machine translation output.