Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

1 Introduction

• Metrics serve as proxies for costly, time-consuming human evaluation and provide immediate development feedback, making their reliability critical to MT research.They are also the primary metric used to report MT system quality.Source: §1 · p. 1
• Correlations reached r = 0.9 across large system cohorts but fell markedly among the few best systems; some metrics had negative correlations for the four best systems.This contrasts with sentence-level evaluation, where metrics better distinguish high-quality from low-quality translations.Source: §1 · p. 1
• Pearson-correlation problems also appeared across a range of system-quality levels, including random samples of systems.The paper tests whether the problems extend beyond small collections of strong systems.Source: §1 · p. 1
• Systems far worse than the rest can disproportionately affect correlations despite offering little insight, and a robust outlier-identification method revealed radically different conclusions about some metrics’ utility.The paper examines this effect using recent years of WMT metric data.Source: §1 · pp. 1–2
• The paper quantifies the metric improvement needed for pairwise rankings to reflect human judgments, accounting for type I errors and type II errors.Type I errors accept negative or insignificant human differences; type II errors reject human-significant differences.Source: §1 · p. 2
• Current metric-evaluation methods can give false confidence, and leading metrics are inadequate substitutes for human evaluation in MT research.They may require untenably large improvements to act as gatekeepers or be used so permissively that good ideas are not rejected.Source: §1 · p. 2

2 Related work

• Since 2007, WMT has held an annual shared task evaluating metrics by their correlation with human judgements across submitted MT systems, and both human evaluation and metric meta-evaluation methods have evolved.Source: §2 · p. 2
• WMT moved from Spearman to Pearson in 2014 because small score differences made Spearman harshly penalize ordering changes; William’s test compares dependent correlations, with unbeaten metrics named winners (Callison-Burch et al., 2006; Bojar et al., 2014; Graham and Baldwin, 2014).Source: §2 · p. 2
• Pearson’s r can be drastically shifted by a single outlier, creating apparent correlations or masking real ones, and different underlying relationships can yield the same coefficient.Source: §2 · p. 2
• BLEU changed little, yet its human-score correlation varied across WMT 2006–2016.Earlier lows may reflect strong rule-based systems, while recent neural-system sets have high correlations for most metrics.Source: §2 · p. 2
• Although other metrics surpassed BLEU at every WMT iteration, it remains the de facto standard for evaluating research hypotheses despite documented limitations.Source: §2 · p. 2

3.1 Direct Assessment (DA)

• DA scores come from WMT 2019 human evaluation, where annotators rate translation adequacy on a 0–100 slider and system scores average standardised translation scores.Bad-quality annotations are filtered using task quality-control items; the Wilcoxon rank-sum test is used to assess whether score differences between systems are due to chance.Source: §3.1 · p. 3

3.2 Metrics

• WMT 2019 metrics compare MT output with human references, while participants could also submit source-based quality-estimation metrics.The analysis focuses on metrics included in the WMT 2019 metrics task.Source: §3.2 · p. 3

Baseline metrics

• BLEU measures brevity-penalised n-gram precision, TER counts edits, and CHRF (Popović, 2015) compares character n-grams.The analysis uses sacreBLEU (Post, 2018) to address BLEU’s sensitivity to hyperparameters and preprocessing; CHRF can match morphological variants.Source: Baseline metrics · p. 3

Best metrics across language pairs

• YISI-1 compares phrase-level semantic similarity using BERT embeddings, while ESIM (Chen et al., 2017) compares sentence representations derived from BERT.Source: Best metrics across language pairs · p. 3

Source-based metric

• YISI-2 uses cross-lingual embeddings to compare MT output with the source, unlike YISI-1, and baseline metrics were designed for multiple references but recently used with one.Source: Source-based metric · p. 3

4.1 Are metrics unreliable when evaluating high-quality MT systems?

• Ma et al. (2019) found declining correlations for seven of 18 language pairs as samples narrowed, but small-sample instability and limited system counts prevent a reliable quality-based conclusion.The authors report that correlations are unstable at small sample sizes and that language pairs typically have only 10–15 MT systems.Source: §4.1 · pp. 3–5
• Correlation does not consistently worsen for stronger systems: German–English and English–German decline, English–Russian stays level, and English–Chinese improves.English–Kazakh also shows the phenomenon despite its top systems being far from human translation; rolling windows show no consistent correlation trend with system quality.Source: §4.1 · pp. 4–5

4.2 How do outliers affect the correlation of MT evaluation metrics?

• An extreme English–German outlier makes every metric’s correlation exceed 0.97, illustrating how Pearson’s r can overstate reliability.When neither outlier is present, system selection has more influence on correlation, and YISI-1 and ESIM usually correlate higher than BLEU.Source: Figure 2 · p. 5, §4.2 · p. 5
• The analysis detects univariate outliers using human ratings and robust median/MAD scores, rather than multivariate outlier removal that could discard genuine metric errors.The method discards systems whose robust-score magnitude exceeds 2.5.Source: §4.2 · p. 5
• Removing outliers drops English–Kazakh BLEU correlation from 0.85 to 0.58, exposes near-zero English–German YISI-2 correlation, and removes English–Russian YISI-2’s apparent association.For English–Russian, YISI-2’s correlation is strongly negative with the low-quality outlier and shows no association once it is removed.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 7

5 Beyond correlation: metric decisions for system pairs

• The analysis compares metric and human significance decisions across WMT 19 system pairs, using significance tests suited to each score type.Human scores use the Wilcoxon rank-sum test; BLEU uses bootstrap testing, YISI-1 and ESIM use paired t-tests, and CHRF is treated as a micro-average to enable a paired t-test.Source: §5 · pp. 7–8
• Figure 4 shows that metric-insignificant bins contain many human-significant pairs, while high metric-difference bins still contain substantial human-insignificant pairs.The comparison covers metric-score differences against human DA differences across MT system pairs.Source: Figure 4 · p. 7, §5 · p. 8
• For BLEU, humans prefer one system in half of insignificant-score pairs; over half of significant 0–3-point pairs are human-indistinguishable, as are about a quarter at 3–5 points.The authors argue that reducing Type I errors would require a threshold below the level needed for the bootstrap significance test, while gatekeeping against Type II errors would require a very large minimum improvement.Source: §5 · p. 8
• The same Type I and Type II error patterns remain when analysis is restricted to researcher-submitted neural systems.Source: §5 · p. 8
• TER can wrongly indicate that a system is much better than another when humans judge them similar or even reach the opposite conclusion.Source: §5 · p. 8
• CHRF, YISI-1, and ESIM make fewer errors than BLEU and TER and should replace BLEU, though their thresholds may remain punitive for gatekeeping or permissive for nurturing progress.When these metrics miss a system difference, the human score difference is considerably lower than when BLEU misses one.Source: §5 · p. 8
• BLEU and TER share 80% of their errors; across the five metrics, over half of each metric’s erroneous decisions are common to all others.When BLEU or TER disagrees with CHRF, ESIM, or YISI-1, BLEU or TER is more likely to be wrong; ESIM’s decisions diverge somewhat more from the other metrics.Source: §5 · pp. 8–9, Figure 5 · p. 8

• Pearson’s correlation is unstable for small samples of similar-quality systems, and strong metric-human correlations can evaporate partly because of sample noise rather than metric shortcomings.The authors call for better methods to test whether metrics are less reliable on high-quality MT systems.Source: §6 · p. 9
• Outlier systems can disproportionately inflate metric-human correlations, even when other systems show no association; removing them widens the correlation gap between BLEU and CHRF, YISI-1, and ESIM.The conclusion recommends removing outliers before computing Pearson’s r, using the technique outlined in Section 4.2.Source: §6 · p. 9
• A single correlation value can conceal different error patterns, so the conclusion recommends visualizing metric against human scores with language-pair scatter plots or the compressed graph in Figure 5.Source: §6 · p. 9
• 1-2 BLEU-point differences correspond to true human-judged quality improvements only about half the time, weakening the rule of thumb for publishing small BLEU gains.The analysis assumes Direct Assessment is a gold standard despite its shortcomings.Source: §6 · p. 9
• The conclusion recommends replacing BLEU or TER with CHRF, YISI-1, or ESIM for evaluating MT systems, while treating human evaluation as the gold standard.It says these alternative metrics are more powerful for assessing empirical improvements, but all automatic metrics are inadequate substitutes for establishing significant gains.Source: §6 · p. 9
• Small changes in automatic metric scores should not be the sole basis for important empirical conclusions; the conclusion calls for supporting them with manual evaluation.Source: §6 · p. 9

A The effect of removing outlier systems on the results of the WMT 19 metrics task

• For non-English language pairs, Table 3 reports Pearson correlations and, where outlier systems occur, correlations after their removal.Bold values indicate metrics not significantly outperformed by another metric for that language pair.Source: Table 3 · p. 12
• For to-English language pairs, Table 4 reports Pearson correlations and, where outlier systems occur, correlations after their removal.Bold values mark metrics not significantly outperformed for that language pair.Source: Table 4 · p. 13
• For from-English language pairs, Table 5 reports metric correlations and, where outlier systems occur, correlations after their removal.Bold values indicate metrics not significantly outperformed by another metric under the Williams Test.Source: Table 5 · p. 14