To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation

1 Introduction

• Metrics can poorly approximate human judgements, respond to specific phenomena, and overlook error severity, potentially misleading MT system development (Zhang and Toral, 2019; Mathur et al., 2020a; Freitag et al., 2021).Human evaluation is often omitted because it can be expensive and time-consuming (Marie et al., 2021).Source: §1 · p. 1
• Much prior metric evaluation studies sentence- or system-level correlations, although practitioners commonly use scores to rank pairs of systems (Mathur et al., 2020a).Such pairwise comparisons inform state-of-the-art claims, architecture choices, publication decisions, and deployment.Source: §1 · p. 1
• The study develops a pairwise system-level evaluation methodology using the largest human-judgement collection reported to that point, released publicly, to assess metric reliability across language pairs and domains.It also examines whether statistical tests increase decision confidence, whether BLEU use may have harmed research decisions, and how earlier findings compare with its results.Source: §1 · pp. 1–2
• The authors recommend pretrained metrics such as COMET, string-based metrics such as ChrF for unsupported languages or as a secondary metric, paired significance tests, and public system outputs; they advise against BLEU.The introduction characterizes BLEU as inferior to other metrics and overused.Source: §1 · p. 2

2.1 Test sets

• Test sets span 101 languages and 232 translation directions, averaging 1,017 sentences per test set.They use authentic news or parliamentary-discussion sources and professionally translated references, checked by a second independent translator; test-set sizes and coverage vary, and most directions are English-centric.Source: §2.1 · p. 2

2.2 Manual quality assessment

• Human assessment uses source-based Direct Assessment, with bilingual annotators rating translations from 0 to 100 without reference translations.Campaigns typically compare two to four systems on around 500 shared sentences; each system pair is evaluated by five annotators. Annotators are paid and quality-controlled, and sentence assignments are balanced across systems and annotators; paired comparisons use the Wilcoxon signed-rank test.Source: §2.2 · pp. 2–3
• Human annotations cover 4,380 systems and 2.3 million sentences; the authors publish the judgements, metadata, metric scores, and replication code.Proprietary test sets and system outputs cannot be released for legal reasons.Source: §2 · p. 2, §2.2 · p. 3

2.3 Systems

• System comparisons comprise model improvements, evaluations against the current best system, and comparisons with publicly available third-party systems.The analyzed models use neural architectures; investigating their varied hyperparameters, training data, and architectures is outside the paper’s scope.Source: §2.3 · p. 3

3 Automatic metrics

• The evaluation focuses on promising and widely used language-agnostic metrics, excluding metrics that support only a small set of languages; Table 1 lists their features.Source: §3 · p. 3
• String-based metrics compare substrings and depend on reference quality, while pretrained metrics use neural models and can handle paraphrases but inherit training-data effects and are harder to diagnose.String-based scores can be easier to interpret because influential substrings can be identified; pretrained metrics can exhibit unexpected behavior or biases learned from training data.Source: §3 · pp. 3–4
• Ma et al., 2019's YiSi metric was unavailable publicly at the time of evaluation.Source: §3 · p. 3
• Metrics used recommended implementations, negative-correlation error metrics such as TER were negated, and evaluation was restricted to language pairs supported by all metrics.Source: §3 · p. 4

4.1 Pairwise score differences

• Prior work mainly correlated individual systems with human judgements, whereas pairwise comparison is argued to better represent general metric use (Mathur et al., 2020a).Earlier studies mostly used Pearson’s correlation and evaluated language directions separately.Source: §4.1 · p. 4
• Pairwise metric evaluation uses score differences between systems and forms comparisons within each campaign, where systems are evaluated under equal conditions.Campaigns comparing two, three, or four systems yield one, three, or six pairs, respectively.Source: §4.1 · p. 4
• All metrics correlate positively with human judgements, but differing language-pair scales make Pearson’s and Spearman’s correlations unsuitable; COMET has the highest correlation overall.COMET’s smallest deviation yields its highest correlation, while COMET, Prism, and especially BLEURT show inconsistent value ranges across language pairs and directions.Source: §4.1 · p. 4

4.2 Pairwise system-level metric quality

• Pairwise accuracy measures how often metric score differences agree in rank with human differences, treating human rankings as gold labels for deciding which system is better.The accuracy is the number of rank agreements divided by the total number of comparisons.Source: §4.2 · p. 5
• Because small human score differences can inflate accuracy by chance, the evaluation also measures accuracy after excluding pairs not distinguished by a Wilcoxon signed-rank test.Accuracy confidence intervals are estimated by bootstrapping; metrics within the best-performing metric’s 95% interval are considered comparable.Source: §4.2 · p. 5

5.1 Which metric is best suited for pairwise comparison?

• COMET achieves the highest pairwise-ranking accuracy, COMET-src is runner-up, and ChrF outperforms BLEU among string-based metrics.Pretrained methods generally beat string-based metrics, except Prism-src.Source: §5.1 · p. 5, Table 2 · p. 5
• Filtering statistically indistinguishable pairs does not change metric ordering, but no metric reaches 100% accuracy even at alpha 0.001; within the outlier-excluded subset, ordering also remains unchanged.The authors infer that metrics cannot fully replace human evaluation or that some remaining systems are incorrectly assessed by annotators; accuracy’s resilience to outliers makes it more suitable than Pearson’s ρ for metric evaluation.Source: §5.1 · pp. 5–6

5.2 Are metrics reliable for non-English languages and other scenarios?

• Across English directions, pretrained metrics retain similar overall ranking: BLEURT is second-best into English, while Prism gains accuracy from English.These results indicate that English-target system pairs do not explain pretrained metrics’ overall advantage over string-based metrics.Source: §5.2 · p. 6
• For non-Latin target scripts, some pretrained metrics drop slightly in rank while ChrF scores higher, suggesting a possible challenge that needs further analysis.Source: §5.2 · p. 7
• Metric rankings remain unchanged on non-WMT news and EU parliamentary discussion data, suggesting COMET is not overfitted to WMT news or languages; string-based accuracy drops in discussion data.The authors speculate that string-based metrics struggle to match disfluent utterances to fluent translations; results across subsets generally show similar metric ordering.Source: §5.2 · p. 7

5.3 Are statistical tests on automatic metric worth it?

• Paired bootstrap resampling at alpha 0.05 increases accuracy by around 10% for all metrics, while 17.3% of COMET’s non-significant pairs are human-significant.This accuracy gain comes with type II errors: the test rejects system pairs as non-significant even though humans judge them significantly different.Source: §5.3 · pp. 7–8, Table 4 · p. 7

5.4 Does BLEU sabotage progress in MT?

• BLEU ranks first for incremental systems, while pretrained metrics generally outperform string-based ones on independent systems, suggesting BLEU-based development rejected human-preferred models.This is indirect evidence; sound conclusions require evaluating rejected systems with other metrics and human judgements. The authors also note that BLEU-preselected systems may be easy for all metrics to differentiate.Source: §5.4 · p. 8, Table 5 · p. 8

• WMT Metric Shared Tasks are the largest source of metric evaluations, while Freitag et al. (2021) raised concerns about crowd annotators’ quality and literal-translation preferences in into-English evaluation.Source: §6 · p. 9
• The Hunter-Schmidt method combines correlations of different sample sizes within each study across language pairs, effectively using a weighted mean to generalize prior findings.Source: §6 · p. 9
• The analysis selected metrics evaluated in multiple studies or promising in a study, using variants closest to recommended settings such as SacreBLEU or settings used by Mathur et al. (2020b).Source: §6 · p. 9
• Pretrained methods outperform string-based methods; ChrF exceeded BLEU in every year, while ESIM performed best overall and COMET performed best after excluding the suspicious system (Mathur et al., 2020b; Ma et al., 2019, 2018).Source: §6 · p. 9
• COMET scores submitted in Mathur et al. (2020b) missed one tokenized English–Chinese system, which strongly skewed Pearson correlation; replication scored it successfully, with no other Chinese issues observed.Source: §6 · p. 9

Discussion

• Pretrained methods outperform string-based methods; the authors recommend COMET as primary and ChrF as secondary and for unsupported languages.They identify ChrF as the best-performing string-based method.Source: §7 · pp. 9–10
• Pretrained metrics may inherit data biases, fail on particular domains, favor fluency over adequacy, or share training data or models with the evaluated system.Their language coverage is limited and quality can vary by language; the authors warn that metric preselection could invite metric-hacking.Source: §7 · p. 10
• COMET-src has high accuracy and enables automatic evaluation on monolingual domain-specific test sets, as suggested by Agrawal et al. (2021).Source: §7 · p. 10
• The authors support deprecating BLEU as the evaluation standard, presenting indirect evidence that its overuse negatively affects MT development.Marie et al. (2021) found that 98.8% of MT papers use BLEU.Source: §7 · p. 10
• Statistical significance tests can improve metric-decision reliability, but their assumptions often fail and they omit random-seed variation, so small metric gains warrant caution.Source: §7 · p. 10
• Publishing translated test-set outputs with papers lets others recalculate scores using different tools or metrics and compare with past and future research.Marie et al. (2021) found that almost 40% of MT papers from 2020 copied scores from other papers without recalculating them.Source: §7 · p. 10
• The analysis has few non-English systems, mostly high-resource language pairs, and many translationese-to-authentic directions, limiting conclusions about other settings.The authors identify non-English pairs, low-resource languages, and the reverse translation direction as future-work areas; Zhang and Toral (2019) found the direction studied is easier for systems to score highly by human judgment.Source: §7 · p. 10
• The analysis assumes human judgments are the gold standard, although annotators or judgment methods may not capture true assessment (Freitag et al., 2021).In direct assessment, humans cannot explicitly mark critical errors and instead usually assign low scores.Source: §7 · p. 10

• Because metrics use different scales across languages, Pearson’s correlation cannot be used; the paper introduces accuracy to evaluate metrics in pairwise system comparisons.Source: §8 · p. 10
• The released human-judgment collection confirms pretrained metrics outperform string-based metrics; COMET is best overall, ChrF is best among string-based methods, and COMET-src may enable monolingual test sets for quality estimation.Source: §8 · p. 10
• The paper reports no metric drawbacks across languages or domains, especially for methods pretrained on human judgment, and presents indirect evidence that BLEU overuse negatively affects MT development.Source: §8 · p. 10
• Statistical testing of automatic metrics largely increases the reliability of pairwise decisions based on metric scores.Source: §8 · p. 10
• The paper endorses publishing translated research-system outputs so that scores can be compared and recalculated in the future.Source: §8 · p. 10

A Metrics Implementation Details

• The metrics generally used common implementations with default or recommended parameters to simulate standard usage.Source: A Metrics Implementation Details · p. 13
• BLEU, ChrF (Popovi´c, 2015), and TER used SacreBLEU 1.5.0, with language-specific tokenizers for Chinese and Japanese.The mteval-v13a tokenizer was used for all other language pairs.Source: A Metrics Implementation Details · p. 13
• CharacTER used commit c4b25cb, EED commit f944adc, and BERTScore (Zhang et al., 2020) version 0.3.7.Source: A Metrics Implementation Details · p. 13
• BLEURT (Sellam et al., 2020) used recommended bleurt-base-128, is fine-tuned for English only, and had a better-performing bleurt-large-512 variant added to Table 8.The implementation was version 0.0.1.Source: A Metrics Implementation Details · p. 13
• COMET (Rei et al., 2020) used wmt-large-da-estimator-1719 and COMET-src used wmt-large-qeestimator-1719; no other evaluated COMET model performed better than the recommended model.The implementation was version 0.0.6.Source: A Metrics Implementation Details · p. 13
• Prism and Prism-src (Thompson and Post, 2020) used commit 06f10da, while ESIM used the mteval-in-context implementation.Source: A Metrics Implementation Details · p. 13

B Confidence Interval for Metric Accuracy

• The confidence interval for the best-performing metric is estimated with modified paired bootstrap resampling, also used for significance testing but differently.The method resamples observations with replacement and calculates accuracy for each resample.Source: B Confidence Interval for Metric… · p. 13
• 10,000 resamples draw system pairs with replacement; each metric’s accuracy is compared with metric α, the best-performing metric on all system pairs.For each resample, the procedure notes which metrics equal or exceed metric α’s accuracy.Source: B Confidence Interval for Metric… · pp. 13–14
• A metric X is considered on par with metric α at 95% significance if α outperforms X in fewer than 95% of resamples.Source: B Confidence Interval for Metric… · p. 14

C Comparing Statistical Tests

• Human system equivalence is defined by Wilcoxon’s test on judgments at α = 0.05, while automatic metrics use paired bootstrap resampling; the tests’ differing power prevents direct comparison.The null hypothesis is that both systems have the same translation quality.Source: C Comparing Statistical Tests · p. 14
• For pairs deemed significantly different by bootstrap resampling, the section measures metric ranking accuracy, including truly different pairs and type I errors, where human and metric rankings may still disagree.Even when both tests find a significant difference, they may disagree about which system is better.Source: C Comparing Statistical Tests · p. 14
• The second scenario examines pairs for which bootstrap fails to reject equal quality despite human judgments finding a difference, while noting that differing test power affects type II errors.The tests’ non-significant outcomes are not directly comparable because their power differs.Source: C Comparing Statistical Tests · p. 14
• Figure 2 plots system-pair differences in human judgments against metric scores, marking ranking disagreements in pink quadrants and distinguishing language-direction groups by color.Spearman’s ρ and Pearson’s r appear on the plot; some BLEU, ChrF, and TER outliers are clipped.Source: Figure 2 · p. 15
• Table 9 reports accuracy by language pair for significantly different system pairs, includes pairs with at least 20 system pairs, and flags Polish–English results as likely post-edited MT output.The caption does not provide the individual accuracy values or identify a best metric.Source: Table 9 · p. 17