Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Papers

1 Introduction

• MT evaluation had no well-defined guideline or clear prerequisites for deciding when comparisons between systems were valid.The extent to which recognized evaluation requirements were met in MT publications was unclear.Source: §1 · p. 1
• Across 2010–2020, an increasing number of papers relied exclusively on BLEU, while most did not perform significance testing, especially after 2016.Papers also increasingly copied BLEU scores from earlier work, while standardized metric-reporting tools remained far from widespread.Source: §1 · p. 1
• The meta-evaluation manually annotated 769 research papers published from 2010 to 2020.Source: §1 · p. 1
• The authors propose a general guideline for automatic MT evaluation and a simple scoring method to meta-evaluate a paper.They suggest adoption could help reverse the concerning trends identified in the meta-evaluation.Source: §1 · p. 2

2 A Survey on MT Evaluation

• The survey annotated 769 papers comparing translation quality between at least two systems, found through title-term searches of *ACL conference publications from 2010 to 2020.The considered main conferences were ACL, NAACL, EACL, EMNLP, CoNLL, and AACL.Source: §2 · p. 2
• Annotations tracked overall-quality automatic metrics, whole-text human evaluation, and whether statistical significance testing of metric-score differences was reported.Metric variants and metrics targeting only specific translation aspects were excluded; targeted error evaluations did not count as whole-text human evaluation.Source: §2 · p. 2
• The survey checked whether automatic evaluation was supported by human evaluation, not for pitfalls within human evaluation itself.Earlier work had studied human-evaluation pitfalls.Source: §2 · p. 2
• The survey also recorded copied metric scores, SacreBLEU use, and whether compared systems used exactly the same preprocessed training, validation, and test data.Source: §2 · p. 2
• A positive annotation for A2–A6 could apply to only one comparison; a SacreBLEU “yes” did not establish consistent use, signature reporting, or valid score comparisons.A6 required checking the data used in previous work, unlike the other annotation questions.Source: §2 · p. 2

3 Pitfalls and Concerning Trends

• The accumulation of exclusive BLEU use, missing significance tests, copied incomparable scores, and different datasets can lead to scientifically dubious MT evaluation.The section examines how often these four pitfalls occur and how their accumulation affects evaluation credibility.Source: §3 · pp. 2–3

3.1 The 99% BLEU

• 98.8% of annotated papers reported BLEU, and 74.3% evaluated systems with BLEU alone, rising to 82.1% in 2019–2020.This persisted despite 108 new metrics proposed in the decade; 89% of those metrics were never used in an *ACL MT publication beyond their proposing papers.Source: §3.1 · p. 3
• For WMT20 Ja→En, BLEU ranked NiuTrans first, 1.1 points above second-ranked Tohoku-AIP-NTT, whereas chrF favored Tohoku-AIP-NTT and better correlates with human judgment.Relying only on these BLEU scores without significance testing or human evaluation would lead to the conclusion that NiuTrans was best.Source: §3.1 · pp. 3–4

3.2 The Disappearing Statistical Significance Testing

• No year exceeded 65.0% of papers performing significance testing, and its use sharply decreased after 2016.For WMT20 Ja→En, NiuTrans was significantly better than Tohoku-AIP-NTT in BLEU but not significantly different in chrF, illustrating that one metric alone can give a different conclusion.Source: §3.2 · pp. 4–5

3.3 The Copied Results

• Nearly 40% of papers in 2019 and 2020 compared their systems with scores copied from previous work, while SacreBLEU remained far from widely adopted.SacreBLEU can standardize metric scores when all compared scores use it, but copied BLEU scores were not always computed with SacreBLEU, even in 2020.Source: §3.3 · p. 5
• BLEU scores from different processing conditions are not comparable; normalization and aggressive tokenization can raise scores several points above original SacreBLEU scores.The paper notes that papers still compare tokenized BLEU scores with scores from other work without knowing how tokenization was performed.Source: §3.3 · p. 6, Table 3 · p. 6

3.4 The Data Approximation

• BLEU scores varied by several points with sentence-length filtering, and truecasing changed them by more than 0.5 points.Because the compared configurations changed multiple variables, their score differences could not establish that truecasing decreased translation quality.Source: §3.4 · p. 7, Table 5 · p. 7
• 38.5% of papers in 2019–2020 drew superiority conclusions while comparing systems using different data.This estimate is a lower bound: the analysis covered copied-score comparisons, excluded specified comparison types, and omitted doubtful cases.Source: §3.4 · pp. 6–7

4.1 Motivation

• 17.4% (38 papers) in 2019–2020 relied exclusively on BLEU differences, with some scores copied, no SacreBLEU or significance testing, and different datasets.Source: §4.1 · p. 8
• Because known pitfalls are increasingly ignored and accumulated, the authors propose a guideline and a scoring method for assessing evaluation trustworthiness.Each “yes” earns one point, up to four; the method is intended for author self-assessment or program committees and includes checking for a better-correlated metric or human evaluation.Source: §4.1 · p. 8, §4.2 · p. 8
• The guideline and scoring method cover only the aspects discussed, so strict adherence can improve evaluation but cannot guarantee a flawless one.Source: §4.1 · p. 8

4.2 The Guideline

• Evaluations should not rely exclusively on BLEU and may use better-correlated metrics or human evaluation, while statistical testing can assess whether score differences are coincidental.Source: §4.2 · p. 8
• Copied metric scores should not be compared unless computed identically with tools that guarantee comparability and with enough information for reproduction.Source: §4.2 · p. 8
• Superiority comparisons require the same preprocessed training, validation, and test data, unless the proposed method or algorithm depends on a particular dataset or preprocessing.Source: §4.2 · p. 8

• The scoring criteria ask whether copied scores are computed with tools that guarantee comparability, such as SacreBLEU.Source: §2 · p. 8
• MT evaluation worsens according to the yearly average meta-evaluation scores shown in Figure 6.Source: Figure 6 · p. 9, §2 · p. 9
• For dataset-independent comparisons, the criteria ask whether systems use exactly the same preprocessed training, validation, and test data.Source: §2 · p. 9

• The accumulated pitfalls and concerning trends identified in annotated papers led the authors to propose a guideline for automatic MT evaluation.Source: §5 · p. 9
• The meta-evaluation does not cover all MT-evaluation pitfalls; papers often use the same language pairs and focus on translation from or into English.The authors also note stronger baseline systems than at the beginning of the decade and plan to extend the meta-evaluation to conferences in machine learning and artificial intelligence.Source: §5 · p. 9
• The authors encourage NLP researchers to conduct similar meta-evaluations in their own areas to uncover pitfalls and concerning trends before they become prevalent.Source: §5 · p. 9