Source-linked AI summary
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
Nitika Mathur, Timothy Baldwin, Trevor Cohn
TL;DR
The paper asks whether automatic MT metrics reliably reflect human judgements when evaluating systems and incremental improvements. It analyzes WMT data, focusing on system selection, outliers, correlation, and pairwise-ranking thresholds. The results show that correlations can create false confidence and that small metric improvements often do not reliably indicate human-improvement, so human evaluation remains necessary.
Problem
Automatic metrics are cheaper and faster than human evaluation, but their reliability as proxies and tools for comparing MT systems is uncertain.
Method
The paper analyzes WMT metric and human-evaluation data across system-quality levels, tests outlier effects, and compares metric-based pairwise thresholds with human significance judgements.
Results
1–2 BLEU points correspond to true human-judged quality improvements only about half the time, while outliers can create falsely high correlations.
Takeaways & Limitations
Metric evaluations should remove outliers and inspect score relationships, while small automatic-metric changes should not be the sole basis for important empirical conclusions.
Takeaways & Limitations
Small samples of similar-quality systems make Pearson correlation unstable, so observed low correlations may partly reflect sampling noise rather than metric shortcomings.
Abstract
from arXiv · showhide
Automatic metrics are fundamental for the development and evaluation of machine translation systems. Judging whether, and to what extent, automatic metrics concur with the gold standard of human evaluation is not a straightforward problem. We show that current methods for judging metrics are highly sensitive to the translations used for assessment, particularly the presence of outliers, which often leads to falsely confident conclusions about a metric's efficacy. Finally, we turn to pairwise system ranking, developing a method for thresholding performance improvement under an automatic metric against human judgements, which allows quantification of type I versus type II errors incurred, i.e., insignificant human differences in system quality that are accepted, and significant human differences that are rejected. Together, these findings suggest improvements to the protocols for metric evaluation and system performance evaluation in machine translation.
1 Introduction
Automatic metrics are central to MT evaluation because they provide cheaper, faster feedback than human assessment, but their reliability depends strongly on which systems are evaluated. The paper investigates outliers, system quality, and pairwise-ranking thresholds to assess when metric-based conclusions reflect human judgements.
- Motivation: Automatic metrics provide immediate, inexpensive feedback and are the primary reported measure of MT system quality.Human evaluation is considerably more expensive and time-consuming, making metric reliability critical to MT research progress.
- Evaluation reliability: High correlations between metrics and humans can fall markedly when evaluation considers only a few of the best systems.Using many systems produced correlations as high as r = 0.9, whereas correlations reduced markedly for the strongest systems.
- Evaluation reliability: Some metrics showed negative correlations among the four best systems, suggesting limited reliability for distinguishing similarly high-quality outputs.The paper contrasts this with more reliable coarse distinctions between poor and good translations.
- Research questions: The study asks whether Pearson-correlation problems extend beyond small sets of strong systems and examines a range of system-quality levels.It includes random samples of systems and reports that the problem is widely apparent.
- Outliers: Outlier systems can disproportionately affect Pearson correlation and produce radically different conclusions about metric utility.The paper proposes a robust method for identifying outliers and testing their effect on correlation.
- Implications: Current metric evaluation can create false confidence, while leading metrics require either very large improvements or permissive thresholds for system comparison.The paper therefore questions whether metrics can substitute for human evaluation in MT research decisions.
2 Related work
Prior WMT metric evaluations changed from Spearman to Pearson correlation and commonly use correlations with human scores to rank metrics. Related work shows that correlation is sensitive to outliers and the evaluated system pool, while BLEU remains standard despite documented limitations.
- WMT evaluation: WMT evaluates automatic metrics by correlating their scores with human judgements across submitted MT systems.The shared task has operated annually since 2007, while both human evaluation and metric meta-evaluation methods have evolved.
- WMT evaluation: Spearman correlation was replaced by Pearson correlation because small score differences between many system pairs were harshly penalized by rank-order disagreements.WMT also uses William’s test to assess whether differences between metric correlations are statistically significant.
- Correlation limitations: Pearson correlation is highly sensitive to outliers and can suggest a strong relationship when none exists or conceal a true relationship.Different underlying relationships can also share the same correlation coefficient.
- Correlation limitations: Metric–human correlation depends on the systems included, with BLEU’s correlation changing across WMT years and language pairs.Earlier low BLEU correlations may have reflected strong rule-based systems receiving low BLEU scores.
- BLEU’s status: BLEU remains the de facto research metric despite being surpassed by other metrics and despite extensive evidence of limitations in BLEU and automatic metrics generally.This persistence motivates closer scrutiny of standard metric-evaluation practice.
3 Data
The study uses WMT 2019 direct-assessment data and examines reference-based and source-based automatic metrics. The metric set includes BLEU, TER, CHRF, YISI-1, ESIM, and YISI-2, which use different surface, edit, character, semantic, and source-comparison strategies.
- Human evaluation data: WMT 2019 direct assessment provides standardized human scores for MT systems, typically using 1,500–2,500 annotations per system where annotator availability was sufficient.Annotators rate adequacy on a 0–100 slider; scores are filtered, standardized per annotator, and averaged by system.
- Automatic metrics: Automatic metrics compare MT outputs with human reference translations, while some submitted metrics instead rely on the source.The analysis focuses on metrics included in the WMT 2019 metrics task.
- Reference-based metrics: BLEU measures reference-based n-gram precision with a brevity penalty, TER counts edits needed to reach the reference, and CHRF compares character n-grams.CHRF’s character-level comparison helps match morphological variants.
- Semantic metrics: YISI-1 computes phrase-level semantic similarity against the reference using contextual word embeddings, while ESIM uses BERT-based sentence representations in a trained neural model.Both metrics compare the output and reference through semantic representations rather than only surface n-grams.
- Source-based metric: YISI-2 computes output–source similarity with cross-lingual embeddings and otherwise follows YISI-1.The baseline metrics were designed for multiple references, although recent practice has used a single reference.
4 Re-examining conclusions of Metrics Task 2019
The paper reexamines metric reliability by testing how system selection, sample size, and outliers affect Pearson correlations with human scores. Results show unstable or misleading conclusions, motivating robust outlier detection and more careful interpretation of correlation.
- 4.1 Are metrics unreliable when evaluating high-quality MT systems?: Correlation depends strongly on the systems sampled, so conclusions about metric reliability cannot be separated from evaluation-set composition.Across language pairs, changing the evaluated systems can decrease, preserve, or increase correlation, including when examining systems near or far from human quality.
- 4.1 Are metrics unreliable when evaluating high-quality MT systems?: Rolling windows keep sample size constant and reveal unstable correlations, especially among small groups of high-quality systems.For German–English, BLEU and CHRF move from perfect correlation for systems ranked 2–5 to −1 for the top four, while ESIM behaves oppositely.
- 4.1 Are metrics unreliable when evaluating high-quality MT systems?: Correlation becomes more stable as N increases, but no consistent quality-dependent trend appears across language pairs.Because WMT language pairs typically contain only 10–15 systems, the authors conclude that these samples are insufficient for deciding whether reliability decreases with system quality.
- 4.2 How do outliers affect the correlation of MT evaluation metrics?: Outliers can radically change correlation, producing falsely high, near-zero, or strongly negative estimates of metric utility.For English–German, YISI-2’s apparent relationship becomes negligible after removing an outlier; for English–Russian, removing a wrongly scored outlier eliminates a strong negative correlation.
- 4.2 How do outliers affect the correlation of MT evaluation metrics?: The authors detect univariate human-score outliers with robust median and MAD statistics, using a cutoff of 2.5 for the robust score.They avoid multivariate removal because metric–human disagreements may be informative about metric errors.
- 4.2 How do outliers affect the correlation of MT evaluation metrics?: For English–Kazakh, BLEU’s correlation with DA drops from 0.85 to 0.58 when outliers are removed.Other language pairs retain high reference-based correlations, while English–German shows a wider metric gap after its two outliers are removed.
5 Beyond correlation: metric decisions for system pairs
The paper evaluates whether metric score differences reliably identify human-significant differences between system pairs, using statistical tests and error analysis. BLEU and TER show substantial mismatches, while CHRF, YISI-1, and ESIM make fewer errors but still require problematic thresholds.
- Pairwise evaluation: The analysis compares human and metric score differences across all WMT 19 system pairs using significance tests tailored to the human and metric scores.Human differences use the Wilcoxon rank-sum test; metric differences use bootstrap, paired t-tests, or a micro-average treatment depending on the metric.
- Pairwise evaluation: Figure 4 shows many human-significant pairs classified as metric-insignificant, while some large metric differences correspond to human-insignificant pairs.“NS” denotes an insignificant metric difference; colours distinguish human-insignificant, significantly worse, and significantly better pairs.
- BLEU and error types: For insignificant BLEU differences, humans judge one system better in half of system pairs, producing Type I errors.These errors include cases where BLEU fails to detect meaningful human differences.
- BLEU and error types: For significant BLEU differences between 0 and 3 points, more than half of system pairs are human-insignificant, while differences of 3–5 points still include about a quarter of similar-quality pairs.These Type II errors make BLEU thresholds unsuitable for reliable gatekeeping without becoming extremely large.
- Other metrics: CHRF, YISI-1, and ESIM make fewer errors than BLEU and TER, but their thresholds remain punitive for gatekeeping or permissive for nurturing progress.When these metrics miss differences, the human score differences are considerably lower than for BLEU.
- Error agreement: BLEU and TER overlap in 80% of their errors, and over half of each metric’s erroneous decisions are shared with all other metrics.ESIM’s decisions diverge somewhat more from the other metrics, but all five metrics exhibit common biases.
6 Conclusion
The conclusion finds that correlation-based metric evaluation is vulnerable to small samples and outliers, while pairwise metric differences often poorly reflect human judgements. It recommends robust correlation analysis, alternatives to BLEU and TER, and manual evaluation for important empirical claims.
- Correlation evaluation: Pearson correlation can be unstable for small sets of similar-quality systems, so strong correlations may partly reflect sample-size noise rather than metric reliability.The paper calls for better methods to test metric reliability on high-quality MT systems.
- Correlation evaluation: Outlier systems can disproportionately raise metric–human correlations and create false confidence, even when no association exists among the remaining systems.The paper recommends recomputing correlations after removing outliers.
- Correlation evaluation: A single correlation value can conceal different error patterns, making scatter plots and compact error visualisations important for assessing metric reliability.The paper points to language-pair scatter plots and Figure 5 as useful visualisations.
- Pairwise system evaluation: A 1–2 point BLEU improvement corresponds to a true human-judged quality improvement only about half the time.The analysis treats Direct Assessment as the gold standard despite its shortcomings.
- Recommendations: The paper recommends replacing BLEU and TER with CHRF, YISI-1, or ESIM for evaluating empirical improvements, while retaining human evaluation as the gold standard.It also advises against using small metric changes alone for important empirical conclusions.
A The effect of removing outlier systems on the results of the WMT 19 metrics task
The appendix tables report Pearson correlations by language-pair grouping and distinguish results before and after removing outlier systems. Bold values mark metrics not significantly outperformed by another metric under the stated comparison test.
- Table 3: Table 3 reports Pearson correlations for language pairs not involving English and adds post-outlier-removal correlations where outliers occur.Bold correlations indicate metrics not significantly outperformed by another metric for that language pair.
- Table 4: Table 4 reports Pearson correlations for to-English language pairs and includes results after removing outlier systems when applicable.
- Significance marking: Bold values identify correlations not significantly outperformed by another metric under the Williams Test.
- Table 5: Table 5 reports correlations for from-English language pairs and likewise shows post-outlier-removal results where applicable.