Source-linked AI summary
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing
Brian Thompson, Matt Post
TL;DR
The paper addresses the breakdown of automatic MT metrics’ correlation with human judgments, especially for BLEU. It proposes a multilingual NMT system as a lexically/syntactically unbiased zero-shot paraphraser for reference-based evaluation and source-based quality estimation, achieving state-of-the-art performance on WMT shared tasks without human-judgment training.
Problem
Existing automatic metrics, especially BLEU, increasingly fail to correlate with human judgments for very strong MT systems.
Method
A multilingual NMT system is used as a lexically/syntactically unbiased paraphraser through zero-shot same-language translation, scoring outputs against references or sources.
Results
The method achieves state-of-the-art performance on WMT shared metrics and quality-estimation tasks without training on prior human judgments.
Takeaways & Limitations
A good multilingual NMT system can serve as a state-of-the-art MT metric while judging state-of-the-art translation systems.
Takeaways & Limitations
System-level correlation results can be unstable because Pearson’s correlation coefficient is unstable in this setting.
Abstract
from arXiv · showhide
We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser's output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages (excluding Gujarati where the model had no training data). We also explore using our model for the task of quality estimation as a metric--conditioning on the source instead of the reference--and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair.
1 Introduction
The paper addresses weakening agreement between automatic MT metrics and human judgments by scoring outputs with a multilingual, reference-conditioned paraphraser. Prism avoids human-judgment training and performs strongly across MT metrics and quality-estimation tasks.
- Motivation: Existing metrics increasingly correlate poorly with human judgments, especially for very strong systems, creating a need for improved evaluation.The paper identifies BLEU as particularly affected and warns that the problem may grow as MT systems improve.
- Approach: Prism force-decodes MT outputs conditioned on human references and uses paraphraser scores to measure how well outputs paraphrase those references.The model represents potential paraphrases implicitly and requires no human quality judgments for training.
- Approach: A multilingual NMT system is used as a lexically/syntactically unbiased paraphraser through zero-shot same-language translation.Its output distribution is centered around copying the input, matching the ideal case where an MT output matches its reference.
- Results: 39 languages are covered by a single model, which outperforms or ties prior metrics on the WMT 2019 segment-level task except Gujarati.Gujarati was excluded because the model had no training data there.
- Results: The method also distinguishes very strong neural systems at system level and significantly outperforms every WMT 2019 quality-estimation submission.Reference conditioning substantially improves performance and allows evaluation of systems that are stronger translators than the multilingual NMT model itself.
2 Related Work
Related work spans overlap, embedding, semantic, paraphrase-based, and generative metrics, while Prism extends bitext-based paraphrasing to the sentence level. It uses multilingual NMT zero-shot translation and favors copying over synthetic paraphrase generation.
- MT metrics: Earlier MT metrics compare outputs with references using token, character, or edit-distance overlap, embeddings, semantic frames, or contextualized representations.Examples include BLEU, NIST, MEANT, LASER, BERT-based methods, and other embedding metrics.
- Paraphrase resources: Bitext-based paraphrase resources have supported phrase-level paraphrase tables and MT metrics that reward reference-consistent word or phrase alternatives.Prism extends this line of work from phrases to entire sentences without enumerating enormous paraphrase sets.
- Multilingual NMT: Multilingual NMT supports transfer to low-resource settings, including zero-shot translation between language pairs absent from training data.Prism treats same-language paraphrasing as a zero-shot translation task.
- Generative paraphrasing: Generative paraphrasing can use natural paraphrase data or synthetic paraphrases created from bitext, but such datasets and procedures differ from Prism’s copying-centered design.The paper contrasts Prism with generative paraphrasers trained on synthetic paraphrases.
- Prism’s design: A well-trained multilingual NMT system tends to generate copies of its input, supporting its use as a lexically/syntactically unbiased paraphraser.This behavior contrasts with the goal of standard generative paraphrasers, which usually seek lexical or syntactic variation.
3 Method
Prism scores MT outputs by force-decoding them with a multilingual sequence-to-sequence paraphraser conditioned on references or sources. It treats paraphrasing as zero-shot translation and combines token probabilities into sequence-level metrics.
- Scoring MT outputs: Prism force-decodes MT outputs conditioned on corresponding human references and uses model probabilities to score how well outputs paraphrase references.The model estimates token probabilities conditioned on prior output tokens and the input sequence.
- Unbiased paraphrasing: A lexically/syntactically unbiased paraphraser has an output distribution centered on copying its input rather than producing deliberately different paraphrases.The design assumes precise lexical choice and word order can convey subtle meaning.
- Unbiased paraphrasing: Prism implements unbiased paraphrasing with multilingual NMT by treating same-language paraphrasing as zero-shot translation, such as Czech to Czech.The model is trained on bitext but applied to an unseen same-language translation direction.
- Reference and source conditioning: Because Prism is multilingual, Prism-src scores MT output conditioned on the source, while Prism-ref scores output conditioned on the human reference.The source-conditioned version is used for quality estimation as a metric.
- Scoring MT outputs: The method considers sequence-level log probability G and average token-level log probability H for combining token probabilities.The final metric uses length-normalized scores derived from these quantities.
- Aggregation and bidirectional scoring: System-level scores are obtained by averaging segment-level scores over all test-set segments.The paper also explores scoring in both directions to penalize information present in one sentence but absent from the other.
4 Experiments
The experiments train and evaluate a large multilingual NMT paraphraser across WMT19 metric and quality-estimation tasks. They test its paraphrasing bias, scoring choices, multilingual design, and comparisons with established baselines.
- Experimental program: The experiments examine zero-shot paraphrasing behavior, token-probability combinations, and WMT18 development results before reporting WMT19 metric and QE performance.The evaluation includes both system-level and segment-level tasks.
- Model and data: The training design targets language-agnostic representations through diverse language pairs, filtering, and language identification checks.LASER filtering addresses alignment noise, while FastText filtering avoids incorrect decoder language tags.
- Model and data: The model is a Transformer with approximately 745M parameters trained to translate between 39 languages.Training uses 99.8M sentence pairs from several multilingual corpora.
- Baselines: Comparisons include WMT19 submissions, BERTscore, BLEURT, LASER, LASER plus a language model, mBART, and an English-only ParaBank 2 paraphraser.The baselines cover embedding, language-model, denoising, and generative-paraphrasing approaches.
- Evaluation protocol: The evaluation reports WMT19 segment-level Kendall’s τ and system-level Pearson correlation, using bootstrap confidence intervals for significance.Non-overlapping 95% confidence intervals identify statistically significant differences.
- Paraphrasing analysis: Figure 2 bins English system-reference pairs by sentBLEU to relate lexical difference to average H(sys|ref), with each bin’s data fraction shown.The analysis compares Prism and ParaBank 2 paraphrasers.
5 Results
Prism performs strongly across multilingual MT evaluation and quality estimation. Reference conditioning improves performance, while source conditioning remains highly competitive without human references.
- Paraphraser behavior: H(sys|ref) increases monotonically with sentBLEU for Prism, while ParaBank 2 is nearly unchanged from sentBLEU 60 to 100.That high-similarity range contains only about 8.5% of system outputs.
- Paraphraser behavior: Prism makes a copy of the input almost as probable as beam-search output, whereas ParaBank 2 prefers its own beam-search output.Prism beam output has BLEU 82.8 against the input, versus 31.9 for ParaBank 2.
- Scoring choices: Length-normalized log probability H slightly outperforms unnormalized log probability G, and equal weighting of H(sys|ref) and H(ref|sys) is approximately optimal.For source conditioning, H(src|sys) adds no useful information to H(sys|src).
- Segment-level metrics: On non-English target directions, Prism significantly outperforms prior work in 7 of 11 language pairs and ties for best in the remainder except Gujarati.Into English, it is statistically tied with the best prior work in every language pair.
- System-level metrics: Prism system-level correlations are positive in every reported language pair, whereas BLEU is negative in 5 language pairs.BERTscore and YiSi-1 variants are each negative in at least two language pairs.
- Comparison caveats: BLEURT has positive correlations for all language pairs into English but is English-only, limiting its multilingual coverage.The paper cautions that Pearson correlation may be unstable in this setting.
- Comparison caveats: System-level results over all submitted MT systems are treated as uninformative because a single weak system can produce high overall correlation even for a poor metric.The paper therefore focuses on the top four submitted systems for its main comparison.
- Quality estimation: Prism-src significantly outperforms every WMT19 QE submission at segment level in every language pair and outperforms or ties at system level.Prism-ref statistically outperforms Prism-src in all but one language pair.
6 Analysis and Discussion
Human references substantially improve Prism’s evaluation performance over source-only scoring in nearly all language pairs, while the model remains effective despite weaker translation ability than leading systems. The analysis also finds benefits from unbiased paraphrasing, fluency signals, and jointly modeling adequacy and fluency.
- Human references: Reference-based Prism statistically outperforms source-based Prism in all but one language pair.The exception is de–cs, the only WMT19 pair with unsupervised systems; in most pairs, Prism translates substantially worse than the best WMT systems.
- Model capability: Prism can judge state-of-the-art MT systems even when its multilingual NMT system is not state of the art at translation.The paper reports this contrast directly and notes that human references enable ranking systems substantially better than Prism itself.
- Human references: Reference conditioning improves correlations into both English and non-English, suggesting Prism’s gains are not simply reference bias.The improvement is statistically significant except for de–cs, despite human judgments using different reference conditions across directions.
- Paraphraser bias: Prism’s lexically/syntactically unbiased paraphraser generally outperforms ParaBank 2, but usually without a statistically significant margin.Bias appears harmful mainly when MT outputs match or nearly match the reference, with greater potential impact for stronger systems or multiple references.
- Fluency: LM scoring outperforms sentenceBLEU at segment-level correlation in 7/10 language pairs to non-English languages, excluding Gujarati.This indicates fluency remains discriminative despite the high fluency of NMT outputs.
- Adequacy, fluency, and training data: Jointly optimizing adequacy and fluency significantly outperforms the contrastive LASER-based method in most language pairs, while Prism also outperforms mBART auto-encoder scoring with less compute.Prism uses 1.3 weeks on 8 V100s versus 2.5 weeks on 256 V100s for mBART.
7 Conclusion and Future Work
The paper concludes that multilingual NMT can function as an unbiased paraphraser for both MT and QE evaluation, while remaining useful beyond sentence-level evaluation.
- A multilingual NMT system can serve as a lexically/syntactically unbiased paraphraser for MT and QE metrics.
- The method achieves state-of-the-art performance on recent WMT metrics and QE tasks without training on prior human judgements.
- A single released model supports 39 languages.
- Human-created references help the evaluation model judge translation systems stronger than the model itself.
- The authors propose extending Prism to paragraph- or document-level evaluation with corresponding multilingual NMT systems.
B Data Details for Replication
The replication setup combines multilingual parallel corpora, filtering and tokenization procedures, several model configurations, and established comparison metrics.
- Training data comes primarily from WikiMatrix, supplemented by Global Voices, EuroParl, SETimes, United Nations, and WMT Kazakh data.
- The corpus applies LASER selection, language-identification filtering, and n-gram-overlap filtering to remove unsuitable sentence pairs.
- Training includes languages with at least 1M examples, resulting in 39 languages.
- The multilingual NMT system uses a shared 64k SentencePiece vocabulary and a large Transformer trained under compute constraints.
- Comparisons include an English-only ParaBank 2 paraphraser, a multilingual language model, mBART, BLEURT, BERTScore, and submitted baseline metrics.
D WMT 2018 (Development set) Results: System-level, Segment-level, and Sweeps
The WMT 2018 development analysis evaluates Prism through score combinations and segment- and system-level comparisons across translation directions.
- The development analysis sweeps linear combinations of scoring directions using reference- versus source-conditioning and alternative probability scores.
- WMT18 results include segment-level evaluations to and from English, plus system-level evaluations to and from English.
- The tables report pairwise-judgment or MT-system counts and identify top methods using overlapping 95% confidence intervals.
E WMT 2019 Metric and QE as Metric Segment-Level Results
The WMT 2019 evaluation reports segment-level MT-metric and QE-as-a-metric results across language directions involving English and non-English pairs.
- The MT-metrics results exclude QE as a metric and cover language pairs into English, from English, and not including English.
- The QE-as-a-metric results cover language pairs into English, from English, and not including English.
- Each table reports segment-level results with pairwise-judgment counts, top-scoring methods, and overlapping 95% confidence intervals.
F WMT 2019 System-Level results for Top 4 Systems
Tables 17–19 report WMT 2019 system-level results for the top four systems judged by humans, grouped by translation direction relative to English. Statistical significance follows the shared-task procedure, which the paper notes appears extremely noisy.
- Coverage: Tables 17–19 cover the top four systems for language pairs into English, from English, and excluding English.The top-four selection is based on human judgments.
- Statistical testing: Statistical significance is reported following the shared task, but the paper notes that it appears extremely noisy.
- Reporting conventions: n denotes the number of machine translation systems evaluated in each language pair.
- Reporting conventions: Bold marks the top-scoring method and methods whose 95% confidence intervals overlap with the top method.
G WMT 2019 Metric and QE as Metric System-Level Results
Tables 20–25 present WMT 2019 system-level results for standard metrics and quality estimation used as a metric. Results are organized by translation direction and include the relevant baselines and submitted systems.
- Metrics: Tables 20–22 report system-level results for metrics excluding quality estimation as a metric, covering language pairs into English, from English, and excluding English.
- Compared systems: The metric tables include all baselines and submitted systems, while the quality-estimation tables include the corresponding submitted systems.
- Quality estimation as a metric: Tables 23–25 report system-level results for quality estimation as a metric across language pairs into English, from English, and excluding English.
- Reporting conventions: n denotes the number of machine translation systems, and bold marks the top-scoring method plus methods whose 95% confidence intervals overlap with it.