Source-linked AI summary
A Call for Clarity in Reporting BLEU Scores
Matt Post
TL;DR
BLEU scores are difficult to compare across machine translation papers because reporting often omits parameterization and preprocessing details. This paper quantifies the resulting variation, identifies user-supplied reference tokenization as the main cause, and recommends metric-internal scoring supported by SacreBLEU.
Problem
BLEU reporting often omits preprocessing, parameterization, and dataset details, making scores difficult to compare across machine translation papers.
Method
The paper quantifies BLEU variation across configurations, identifies its main sources, and provides SacreBLEU to standardize reference handling and record scoring parameters.
Results
User-supplied reference tokenization is identified as the main cause of BLEU-score incompatibility, with resulting variances often exceeding reported gains.
Takeaways & Limitations
Researchers should report BLEU using metric-internal reference tokenization and explicitly state the parameterization, preferably following the WMT processing scheme.
Abstract
from arXiv · showhide
The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SacreBLEU, to facilitate this.
1 Introduction
BLEU is the dominant machine-translation metric, but its parameterization, preprocessing, and often-unreported settings make scores difficult to compare or replicate across papers. The paper quantifies this incompatibility, identifies user-supplied reference tokenization as its main cause, and proposes metric-supplied tokenization supported by SacreBLEU.
- BLEU’s language independence, computational ease, and correlation with human judgments have made it the dominant metric in machine translation research.
- BLEU is parameterized rather than a single metric, and preprocessing substantially affects scores, making differently processed references incomparable.
- Unreported or difficult-to-discover parameters and preprocessing schemes impede comparison and replication of BLEU scores across papers.
- The paper quantifies these problems, finding variances larger than many reported gains and identifying user-supplied reference tokenization as the main incompatibility.
- SacreBLEU downloads and stores common test-set references and reports a parameter-recording version string to support reproducible BLEU reporting.
2 Problem Description
BLEU is a family of parameterized methods, and preprocessing—especially reference tokenization—can substantially change scores. These undocumented choices prevent direct comparison of published results and complicate reproducible evaluation.
- BLEU parameters: BLEU comprises parameterized methods differing in reference count, length-penalty computation, maximum n-gram length, and smoothing.In practice, most evaluations use one reference, four-gram maximum length, and rarely encounter zero counts.
- Preprocessing: Reference preprocessing changes the n-grams used by BLEU, with tokenization among the most important preprocessing choices.Preprocessing can include normalization, punctuation splitting, compound splitting, and case removal.
- Preprocessing: 1.8-point differences can arise across common reference-tokenization schemes for one language arc, averaging around 1.0.The largest variation comes from case treatment, yet papers often do not state whether BLEU is cased or case-insensitive.
- Preprocessing risks: Reference preprocessing can artificially inflate BLEU when unknown-word masking is applied to references, making pipeline mistakes easy to introduce.Such masking was used by some systems to address limited vocabularies before sub-word splitting became common.
- Comparability: User-supplied reference processing prevents direct comparison of published BLEU numbers and forces readers to reconstruct scattered implementation details.Even meticulous researchers may find it unwieldy to report enough technical detail, leaving uncertainty and extra work for readers.
- Dataset management: WMT’14 English→German results are ambiguous because two datasets exist, one containing 2,737 sentences after roughly 10% were removed and another restoring them.The distinction concerns an evaluation release and a later release, complicating dataset management and result interpretation.
3 A way forward
The parsing community enabled reliable cross-paper comparisons by standardizing evaluation through evalb and its parameter file. For machine translation, SacreBLEU adopts metric-internal preprocessing and documents evaluation settings to reproduce WMT BLEU scores.
- A way forward: PARSEVAL combines precision and recall over labeled parser spans into the F1 metric commonly compared across parsing papers.The comparison uses parser-output spans against gold-standard spans from the correct parse tree.
- A way forward: The parsing community resolved hidden scoring choices by adopting evalb, whose parameter file specifies treatment of ROOT nodes, -NONE- nodes, punctuation, and equivalent labels.This standardization facilitated nearly thirty years of comparisons on treebanks such as the Wall Street Journal portion of the Penn Treebank.
- A way forward: Moses’ multi-bleu.perl and MultEval are unsuitable because they require or advocate user-supplied preprocessing, whereas mteval-v13a.pl uses metric-internal preprocessing and underlies annual WMT evaluations.The WMT-oriented script is presented as a candidate despite an inconvenience involving its data requirements.
- A way forward: SACREBLEU expects detokenized outputs, applies its own preprocessing, produces WMT-equivalent values, automatically supplies test-set references, and records settings in a short version string.Its version string documents parameters such as language pair, tokenization, dataset, smoothing, and software version.
4 Summary
BLEU scores across machine-translation test sets are difficult to compare because processing differences create large score variances. The paper proposes metric-internal reference processing, explicit parameter reporting, and WMT-compatible tooling to make scores directly comparable.
- Large score variances make BLEU results across machine-translation test sets difficult to compare directly.The test sets come from many language arcs and from academic, government, and industry sources.
- Researchers should report BLEU with metric-internal reference tokenization and preprocessing and explicitly state the metric parameterization.These practices would allow scores to be directly compared.
- The paper recommends WMT’s processing scheme for backwards compatibility and provides a tool to facilitate its use.