Source-linked AI summary
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger
TL;DR
Text-generation evaluation needs metrics that judge semantic content rather than surface overlap and correlate well with human quality judgments. MoverScore combines contextualized representations with Earth Mover’s Distance and performs strongly across four generation tasks, while the authors identify costly human references as a future limitation.
Problem
Text-generation systems need a holistic evaluation score based on semantic content rather than surface forms, with strong correlation to human judgments.
Method
MoverScore compares system and reference texts using contextualized representations and Earth Mover’s Distance to measure semantic distance.
Results
MoverScore demonstrates strong generalization across four text-generation tasks and often outperforms supervised metrics.
Takeaways & Limitations
Combining contextualized embeddings with a distance measure is a promising direction for holistic, more human-like text-generation evaluation.
Takeaways & Limitations
The current evaluation requires costly human references; future work aims to score outputs using source texts and system predictions alone.
Abstract
from arXiv · showhide
A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.
1 Introduction
MoverScore addresses limitations of surface-overlap metrics by comparing system and reference texts through semantic representations and distance. Across four generation tasks, it shows strong alignment with human judgments and competitive performance.
- Motivation: Existing metrics such as BLEU and ROUGE rely primarily on n-gram co-occurrence, inadequately covering copying, unseen-word generation, and diverse wording.These limitations motivate more unbiased evaluation for increasingly flexible neural generation systems.
- Goal: MoverScore aims to assign one holistic content-matching score by comparing system outputs with human references according to conveyed meaning.The score should decrease when outputs add or omit content or fail to deliver the intended meaning.
- Approach: The metric combines contextualized representations with Earth Mover’s Distance to quantify semantic distance between system and reference texts.The approach investigates distributional representations for semantic comparison across multiple generation tasks.
- Evidence: MoverScore achieves high correlation with human judgments of text quality.The paper formulates evaluation as semantic-distance measurement using continuous representations that can encode semantic and syntactic deviations.
- Evidence: MoverScore outperforms or performs comparably to strong baselines across summarization, machine translation, image captioning, and data-to-text generation.This cross-task result is presented as evidence that the metric is a promising direction for evaluation.
2 Related Work
Related evaluation metrics often emphasize lexical or manually extracted matches, limiting their ability to recognize semantically equivalent outputs with different surface forms. MoverScore extends semantic evaluation through contextual representations and transport-based distance.
- Evaluation scope: NLG evaluation compares system texts with references across summarization, translation, data-to-text generation, and image captioning.The paper emphasizes metrics applicable across tasks producing sentence- or paragraph-length outputs.
- Existing metrics: ROUGE measures lexical overlap, while Pyramid and BE match content units that may require manual extraction.These metrics can correlate well with human judgments but are not general enough for abstractive paraphrases.
- Existing metrics: Machine-translation metrics commonly use word or character n-gram overlap and edit distance rather than the meanings conveyed.The paper also notes that preprocessing choices can produce inconsistent results for word-based metrics.
- Existing metrics: Data-to-text metrics can poorly evaluate outputs that communicate the same information through lexical and syntactic variation.Prior work found that many word- and grammar-based metrics only weakly reflect human judgments for end-to-end systems.
- Semantic metrics: Recent work replaces lexical overlap with static embeddings, classifiers, or contextualized representations to estimate semantic similarity.Related approaches include BERT- and ELMo-based metrics for translation, summarization, and essay scoring.
- MoverScore’s position: MoverScore investigates embedding granularity, pretrained representations, fine-tuning, layer aggregation, and cross-task generalization through word- and sentence-mover variants.Its stated novelty is the systematic combination of these design dimensions with Word Mover’s Distance.
3 Our MoverScore Meric
MoverScore measures semantic distance by transporting weighted n-gram representations between system and reference texts. It combines contextualized embeddings, aggregation choices, and soft alignments to support flexible comparison across granularities.
- Metric formulation: MoverScore combines contextualized representations with a distance measure that captures both shared content and deviations from the reference.This contrasts with similarity measures focused mainly on intersecting content.
- Semantic distance: N-gram embeddings are computed from word embeddings, with IDF-based weights downplaying less informative function-word n-grams.The method evaluates unigrams, bigrams, and full sentences; full-sentence granularity reduces WMD to Sentence Mover’s Distance.
- Semantic distance: WMD generalizes to weighted n-grams by minimizing transportation cost between system and reference distributions.The flow matrix specifies how much mass moves from each system n-gram to each reference n-gram.
- Representations: Euclidean distance compares n-gram embedding vectors, while contextualized ELMo and BERT embeddings encode whole-sentence information into word vectors.Static word2vec embeddings are used as an alternative but do not capture word order or compositionality as contextualized embeddings do.
- Alignments: MoverScore uses soft many-to-one alignments, unlike BERTScore’s hard one-to-one alignments, by solving a constrained minimum-effort transport problem.An asymmetric transportation cost matrix can bias the metric toward precision or recall.
- Contextualized representations: Contextualized encoders produce layer-specific word vectors that are aggregated into one representation per word.The paper studies power-mean concatenation and routing; routing is relegated to the appendix because it does not improve results over power means.
- Experimental dimensions: The experiments vary n-gram granularity, pretrained embedding mechanism, BERT fine-tuning task, and aggregation technique.BERT is fine-tuned on MultiNLI, QANLI, or QQP, while comparisons include word2vec, ELMo, and BERT.
4 Empirical Evaluation
The evaluation compares semantic metrics built from word- or sentence-level representations against strong task-specific baselines across four text-generation tasks. MoverScore combines embedding choices, fine-tuning, aggregation, and distance-based comparison to assess correlation with human judgments.
- The evaluation measures Pearson and Spearman correlations between metrics and human judgments across machine translation, summarization, image captioning, and dialogue generation.
- MoverScore includes word mover and sentence mover variants that differ in the granularity of their semantic representations.
- Word mover configurations encode text using a specified granularity, embedding source, fine-tuning task, and aggregation method.
- The experiments compare MoverScore with task-specific lexical and supervised baselines, along with BERTScore across all tasks.
- WMD-1/2+BERT+MNLI+PMeans achieves the best correlation across all seven WMT17 language pairs and outperforms the supervised metric RUSE.
4.2 Text Summarization
The evaluation compares summary systems on TAC-2008 and TAC-2009 using human Pyramid and Responsiveness judgments. Lexical metrics correlate above moderately on these datasets, while contextualized word mover metrics perform better or comparably across the reported comparisons.
- Datasets and human judgments: TAC-2008 and TAC-2009 contain 48 and 44 news-article clusters, respectively, with four reference summaries and 57 or 55 system summaries per cluster.Each cluster covers 10 articles, and every summary is shorter than 100 words.
- Datasets and human judgments: Human evaluation uses Pyramid scores for covered semantic content units and Responsiveness scores for summary quality.
- Results: Lexical metrics such as ROUGE correlate above moderately with human judgments on TAC-2008 and TAC-2009.The passage contrasts this result with poorer performance on dialogue generation and image captioning.
- Results: WMD-1+BERT+MNLI+PMeans performs better than or comes close to the strongest comparison metrics in the reported summarization results.
- Cross-task comparison: On task-oriented dialogue, no metric reaches even moderate correlation, although contextualized word mover metrics still achieve the best correlations.The authors speculate that contextualizers poorly represent named entities and numbers in system and reference texts.
Results
Across machine translation, dialogue, and image-captioning analyses, MoverScore generally correlates more strongly with human judgments than unsupervised baselines, while soft alignment is comparable to or slightly better than hard alignment.
- Word mover metrics outperform all baselines except the supervised metric on MSCOCO image-captioning evaluation.
- Hard and Soft Alignments: MoverScore outperforms both asymmetric hard-alignment factors, while harmonic-mean BERTScore is on par with MoverScore in machine translation.Both metrics use representations from BERT’s ninth layer for fair comparison.
- Distribution of Scores: BERT-based word mover metrics distinguish translations at two polar quality levels, whereas lexical metrics struggle with high-quality outputs.SENTBLEU assigns lower scores to some high-quality translations despite correctly identifying low-quality ones.
- Correlation Analysis: 0.6-0.7 correlations with human judgments are achieved by MoverScore metrics, compared with 0.4-0.5 for existing machine-translation metrics.The MoverScore results remain robust across different language pairs.
- Impact of Fine-tuning Tasks: Fine-tuning BERT on MNLI improves average Pearson correlation by 1.8 points across seven language pairs.The analysis compares word mover metrics using BERT fine-tuned on three tasks.
4.6 Discussions
The paper’s strongest results support combining contextualized embeddings with Earth Mover’s Distance for evaluation across diverse text-generation tasks.
- MoverScore outperforms strong unsupervised metrics on three of four tasks and rivals or exceeds state-of-the-art supervised metrics across tasks.The reported unsupervised gains are 5.7 points over METEOR++ in machine translation, 3.0 over SPICE in image captioning, and 2.2 over METEOR in dialogue response generation.
- 74.3 average correlation in machine translation exceeds the state-of-the-art supervised metric by 5.8 points and contemporaneous BERTScore by 2.4 points.
- Contextualized BERT embeddings and fine-tuning on large NLI datasets provide the major improvements over word2vec and ELMo.
- Soft alignments marginally outperform hard alignments, while unigrams slightly outperform bigrams on average in word mover metrics.
- P-means and routing are suggested as effective techniques for consolidating representations across layers.
5 Conclusion
The paper develops unsupervised text-generation metrics that combine contextualized embeddings with Earth Mover’s Distance, with strong generalization across four tasks. It identifies a promising direction toward holistic, more human-like evaluation while noting future work to remove costly human references.
- Contextualized embeddings combined with Earth Mover’s Distance form the basis of the investigated unsupervised evaluation metrics.
- The word mover variant demonstrated strong generalization across four text-generation tasks and often outperformed supervised metrics.
- The metric provides a promising direction toward holistic and more human-like evaluation of text-generation systems.
- Future work aims to evaluate using source texts and system predictions alone, avoiding costly human references.
A.1 Proof of Prop. 1
This proof connects BERTScore to Mover Distance by expressing its precision and recall components in quasi-WMD form. The resulting F-score is the harmonic mean of the corresponding precision- and recall-based WMD forms.
- The section proves that BERTScore precision and recall can be viewed as a non-optimized Mover Distance.
- The WMD formulation represents texts using vectors of weights assigned to each n-gram.
- BERTScore is defined and its recall component is reformulated in a quasi-WMD form.
- The BERTScore F-score is the harmonic mean of the quasi-WMD forms for precision and recall.
A.2 Routing
The routing method aggregates contextualized representations using nonparametric KDE clustering, with iterative updates controlled by a step-size hyperparameter. Layer aggregation performs competitively, while word-mover metrics using fine-tuned BERT representations correlate strongly with human judgments across evaluated tasks.
- Routing: KDE-based nonparametric clustering is introduced for routing because it bridges kernel functions with empirical distributions and can improve computational efficiency.
- Routing: The routing objective uses a distance function, representation closeness weights, and a kernel function to aggregate contextualized layers.
- Routing: The routing process updates representation-assignment weights iteratively with SGD, using α to control the step size.
- Layer aggregation: Word-mover metrics with p-means or routing approach the oracle performance obtained from the best layer, whose identity is task-dependent.
- Experiments: Word-mover metrics using BERT fine-tuned on MNLI achieved the highest human correlations in the reported machine-translation, summarization, and dialogue experiments.