Source-linked AI summary
BLEU might be Guilty but References are not Innocent
Markus Freitag, David Grangier, Isaac Caswell
TL;DR
The paper addresses unreliable automatic MT evaluation by examining how reference quality affects metric–human correlation. It compares reference collection strategies and introduces human paraphrasing to diversify references. Paraphrased references produce more reliable evaluations across metrics and high-quality systems, while standard multi-reference BLEU does not improve correlation; the study also identifies practical and evaluation-scope limitations.
Problem
Automated metrics can disagree with human judgments for high-quality MT, while translationese-heavy references may favor outputs sharing their style.
Method
The paper compares reference collection methods, evaluates metric–human correlations across systems and metrics, and creates diverse human paraphrases of existing references.
Results
Paraphrased references yield stronger human-evaluation correlations across metrics, including BLEU, chrF, METEOR, BERTScore, and Yisi, while multi-reference BLEU does not improve correlation over single-reference BLEU.
Takeaways & Limitations
A single paraphrased reference is suggested for more reliable automatic evaluation, while releasing diverse references enables reassessment of high-quality systems and alternative output styles.
Takeaways & Limitations
Human raters may find paraphrased translations harder to assess, and human evaluation may be less reliable for complex translations.
Abstract
from arXiv · showhide
The quality of automatic metrics for machine translation has been increasingly called into question, especially for high-quality systems. This paper demonstrates that, while choice of metric is important, the nature of the references is also critical. We study different methods to collect references and compare their value in automated evaluation by reporting correlation with human evaluation for a variety of systems and metrics. Motivated by the finding that typical references exhibit poor diversity, concentrating around translationese language, we develop a paraphrasing task for linguists to perform on existing reference translations, which counteracts this bias. Our method yields higher correlation with human judgment not only for the submissions of WMT 2019 English to German, but also for Back-translation and APE augmented MT output, which have been shown to have low correlation with automatic metrics using standard references. We demonstrate that our methodology improves correlation with all modern evaluation metrics we look at, including embedding-based methods. To complete this picture, we reveal that multi-reference BLEU does not improve the correlation for high quality output, and present an alternative multi-reference formulation that is more effective.
1 Introduction
As machine translation improves, automated metrics increasingly diverge from human judgments, while reference translations can bias evaluation toward translationese. The paper studies reference collection and proposes paraphrased references to improve metric–human correlation.
- Recent MT quality gains have raised doubts about automated metrics, especially because top systems can differ between automated and human evaluation.
- Automated evaluation remains essential for architecture selection, hyperparameter search, and data filtering because human evaluation cannot match its pace and scale.
- Typical references concentrate around translationese, causing overlap-based metrics to favor outputs with similar source-influenced style and undervalue equally accurate alternatives.
- Carefully paraphrased references increase diversity, reduce rewards for translation artifacts, and improve correlation with human judgments across metrics and MT systems.
- The paper contributes diverse human paraphrases, an alternative multi-reference formulation, and released references intended to support evaluation of broader output styles.
2 Related Work
Related work establishes automated evaluation as a correlation problem and identifies translationese as a source of mismatch between metric scores and human judgments. The paper responds by using human paraphrases rather than relying on automatic paraphrase generation or narrowly matched references.
- Automatic evaluation compares system outputs with human references using metrics based on overlap, edits, synonyms, contextual embeddings, or multiple alternative formulations.
- Meta-evaluation asks whether automatic scores correlate with human assessments, with prior work finding stronger correlation among similar systems than across different system types.
- Benchmark studies report mismatches between systems with the highest BLEU scores and those judged best by humans, including systems trained with back-translation.
- Translationese can make overlap metrics reward source-influenced output over more natural target-language output, motivating diverse references.
- The paper uses human-generated paraphrases to reduce reference-style bias, whereas automatic paraphrasing methods often produce mainly local changes and remain imperfect.
3 Collecting High Quality and Diverse References
The study constructs additional references through professional translation and human paraphrasing, targeting greater diversity and less translationese. Linguists are instructed to reformulate translations extensively, while adequacy ratings help select strong paraphrases.
- The authors collect two reference types: an additional professional translation and paraphrases of existing references produced by a different set of linguists.
- Professional linguists produce additional references under controlled conditions that prohibit CAT tools and copying from the source field.
- Shuffling sentences across linguists reduces systematic document-level biases while enabling multi-reference evaluation.
- Translationese reflects source-language interference in word order, sentence structure, and lexical choice, and can make reference sets favor similarly affected MT output.
- After minor initial changes, the instructions require linguists to paraphrase as much as possible without changing meaning.
- Examples contrast a typical translation with two paraphrases that differ substantially in sentence structure and from one another.
- Adequacy ratings select the most adequate paraphrase among alternatives, producing a composite high-paraphrase set.
4 Experimental Set-up
The experiments compare reference types on WMT 2019 English→German outputs and evaluate metric–human correlations using campaign human ratings. Human judgments use direct assessments averaged over three ratings per sentence.
- The study evaluates different references on official WMT 2019 English→German submissions and correlates automatic scores with human ratings from the same campaign.
- English→German is selected because its evaluation track contains the most submissions and outputs with the highest adequacy ratings.
- Human raters score adequacy on an absolute 0-100 scale, with three ratings per sentence averaged into the final sentence score.
5 Experiments
The experiments compare reference-collection methods and evaluation metrics against human judgments on WMT English→German. Paraphrased references generally improve system-ranking correlation, especially for high-quality outputs, while multi-reference BLEU does not provide the same benefit.
- 5.1 Human Evaluation of References: The composite paraphrased reference HQ(P) receives a higher human rating than WMT or AR alone, but HQ(P) and the other combined references do not surpass WMT.p in correlation.This challenges the assumption that higher reference adequacy necessarily yields more reliable automatic scores.
- 5.2 Correlation with Human Judgement: Three additional references—AR, WMT.p, and AR.p—show higher BLEU correlation with human judgment than the original WMT reference on all 22 submissions.WMT.p achieves the highest correlation among the individual references.
- 5.2 Correlation with Human Judgement: Multi-reference BLEU does not correlate better with human judgments than single-reference BLEU or the composed HQ(x) references.Combining references that share translationese biases can continue to favor translationese outputs, while adding paraphrases changes n-gram matches only modestly.
- 5.2 Correlation with Human Judgement: Paraphrased references outperform their corresponding original references, with the advantage especially visible when evaluating high-quality system output.The top-k analysis examines the regime where standard-reference correlations are known to break down.
- 5.2 Correlation with Human Judgement: Paraphrased BLEU scores are much lower than standard-reference scores, yet the resulting system rankings achieve the highest correlation among the references examined.The lower scores reflect fewer n-gram matches rather than poorer ranking quality.
- 5.3 Alternative Metrics: Across BLEU, TER, METEOR, chrF, BERTScore, and Yisi-1, paraphrased references generally yield higher human-evaluation correlation than corresponding original references.The only reported exception is TER for HQ(P); Yisi benefits from the higher accuracy of that reference.
6 Why Paraphrases?
Back-translation and APE systems produce less translationese output, causing standard translationese references to underestimate their quality and misorder systems relative to human judgments.
- Why Paraphrases?: Back-translation and APE systems generate less translationese output, so BLEU with translationese references underestimates their quality.These systems use additional target-side monolingual data and are rated higher by humans than the bitext-only system.
- Why Paraphrases?: Human ratings rank the APE and back-translation systems above the bitext-only system.The comparison concerns three systems evaluated on WMT newstest 2019.
- Why Paraphrases?: All human-translated reference sets produce negative BLEU correlation with human ratings and the wrong system order in these extreme cases.The reference sets are WMT, HQ(R), and HQ(all 4).
- Why Paraphrases?: The table evaluates BLEU scores for bitext-trained, back-translation-augmented, and APE-naturalized systems, with correctness indicating agreement with human rankings.This is the evaluation setup described in the table caption.
- Why Paraphrases?: Reference methodology may systematically bias evaluation against modeling techniques that improve human-judged quality.The authors raise whether previous research may have discarded approaches that actually improved MT quality.
7 Characterizing Paraphrases
The paper characterizes paraphrases as more diverse alternatives to monotonic translationese: they lower generic structural overlap while preserving semantically meaningful matches.
- Characterizing Paraphrases: Alignment distance is used as a proxy for word-by-word, or monotonic, translation, with lower scores indicating more monotonic translation.The measure is computed from fast-align alignments across reference translations.
- Characterizing Paraphrases: Paraphrased references have lower BLEU scores than standard references but higher correlation with human judgment.This motivates examining which n-grams overlap with each reference type.
- Characterizing Paraphrases: Standard-reference overlap is dominated by generic common n-grams associated with matching the source sentence structure.These matches are described as crucial for reaching high > 40 BLEU scores.
- Characterizing Paraphrases: Paraphrased-reference overlap more often rewards n-grams expressing the sentence’s semantic meaning.The paper contrasts these matches with the structurally repetitive matches found using standard references.
- Characterizing Paraphrases: Examples of high-overlap 4-grams with paraphrased references include named entities such as “Wheeling, West Virginia” and “Christine Blasey Ford.”Each listed example appears three times among the reported matches.
8 Conclusions
The study finds that paraphrased references make automatic MT evaluation more reliable across metrics and high-quality systems, while highlighting cost and human-rating constraints.
- Conclusions: Paraphrased references yield stronger human correlations across BLEU, chrF, METEOR, BERTScore, and Yisi for WMT 2019 English→German.They especially correlate well for top WMT submissions and distinguish systems augmented with back-translation or APE.
- Conclusions: Standard multi-reference BLEU does not correlate better with human judgments than single-reference BLEU.Selecting the best-rated reference per sentence increases correlation for standard references instead.
- Conclusions: A single paraphrased reference is recommended for more reliable evaluation because combined paraphrased references cost over twice as much to construct.Combined paraphrased references show slightly higher correlation for embedding-based metrics.
- Conclusions: Releasing all reference translations could enable researchers to revisit decisions and measure differences among high-quality or more fluent systems.The paper connects this recommendation to possible bias in standard references against such modeling techniques.
- Conclusions: Paraphrased translations are harder for human raters to assess than translationese sentences, motivating future work on finer ratings and document-level evaluation.The authors suspect human evaluation is also less reliable for complex translations.