Source-linked AI summary
Better Summarization Evaluation with Word Embeddings for ROUGE
Jun-Ping Ng, Viktoria Abrecht
TL;DR
ROUGE’s lexical-overlap bias limits its suitability for abstractive summaries with substantial paraphrasing. The paper incorporates word embeddings into ROUGE to measure semantic similarity, and experiments show better correlations with human judgements under Spearman and Kendall rank coefficients. The evaluation uses TAC AESOP data, whose summaries are strongly biased toward extractive systems.
Problem
ROUGE favors lexical similarities, limiting its suitability for evaluating abstractive summaries with substantial paraphrasing.
Method
The paper incorporates word embeddings into ROUGE to compare semantic similarities between words in generated and human-written summaries.
Results
ROUGE-WE shows very good correlations with human assessments under Spearman and Kendall rank coefficients, with ROUGE-WE-1 consistently outperforming leading state-of-the-art systems.
Takeaways & Limitations
Word embeddings add a promising dimension to summarization evaluation and may expand ROUGE’s applicability to abstractive summarization.
Takeaways & Limitations
The TAC AESOP summaries are strongly biased toward extractive summarizers, motivating evaluation on summaries with substantial paraphrasing.
Abstract
from arXiv · showhide
ROUGE is a widely adopted, automatic evaluation measure for text summarization. While it has been shown to correlate well with human judgements, it is biased towards surface lexical similarities. This makes it unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing. We study the effectiveness of word embeddings to overcome this disadvantage of ROUGE. Specifically, instead of measuring lexical overlaps, word embeddings are used to compute the semantic similarity of the words used in summaries instead. Our experimental results show that our proposal is able to achieve better correlations with human judgements when measured with the Spearman and Kendall rank coefficients.
1 Introduction
ROUGE scales summarization evaluation through lexical overlap, but this favors surface matches and struggles with paraphrased summaries. The paper proposes word embeddings to compare semantic similarity and reports better human-judgement correlations.
- ROUGE was introduced to make summarization evaluation more scalable by comparing overlapping n-grams, word sequences, and word pairs with human-written summaries.
- ROUGE favors lexical similarity, making it unsuitable for abstractive summaries with substantial paraphrasing.
- Word embeddings map words into a vector space where distance corresponds to semantic similarity, enabling ROUGE to compare meaning rather than only lexical form.
- The paper uses word embeddings to reward semantically equivalent phrases even when their lexical representations differ.
- Word embeddings produce better correlations with human judgements under Spearman and Kendall rank coefficients.
2 Related Work
Related evaluation methods include content-unit, semi-automated pyramid, graph-based, and AESOP systems. These approaches vary in automation, granularity, and reliance on human intervention.
- Basic Elements evaluates content completeness by decomposing sentences into smaller information units.
- The pyramid method maps generated-summary content to manually identified Summary Content Units, although automated variants still require manual SCU identification.
- AESOP included multiple summarization evaluation systems between 2009 and 2011.
- AutoSumENG scores summaries by comparing graph structures between generated and model summaries.
3 Methodology
The methodology replaces ROUGE’s lexical matching with word-embedding similarity while retaining ROUGE variants for unigrams, bigrams, and skip-bigrams. It uses word2vec and composes n-gram representations multiplicatively.
- Word2vec maps words or word sequences into vectors so semantically similar words have closer projections.
- ROUGE-1, ROUGE-2, and ROUGE-SU4 measure unigram, bigram, and skip-bigram overlap, respectively.
- The proposal, called ROUGE-WE, defines a new similarity function for incorporating word embeddings into ROUGE.
- ROUGE-WE compares word vectors using their dot product when both words have embeddings, and assigns zero similarity when either word is out of vocabulary.
- The resulting ROUGE-WE variants correspond to ROUGE-1, ROUGE-2, and ROUGE-SU4.
- Individual token embeddings are multiplied element-wise to compose a vector representation for an n-gram.
4 Experiments
The experiments evaluate ROUGE-WE variants on the TAC AESOP dataset by correlating automatic scores with human assessments across pyramid, responsiveness, and readability. Results generally favor word-embedding enhancements for rank correlations, while performance varies by ROUGE variant and correlation metric.
- 4.1 Dataset and Metrics: The TAC AESOP dataset contains 44 topics, four human model summaries per topic, and summaries from 51 participating systems.These summaries form the basis for evaluating automatic metrics against human assessments.
- 4.1 Dataset and Metrics: Human judges assessed summaries using pyramid, responsiveness, and readability, while automatic metrics were compared through Pearson, Spearman, and Kendall correlations.Responsiveness covers requested information and linguistic quality; readability measures fluency and readability.
- 4.2 Results: ROUGE-WE-1 correlates very well with pyramid, responsiveness, and readability scores under Spearman and Kendall rank correlation, although ROUGE-SU4 performs better under Pearson correlation.The comparison covers the three human-assessment dimensions and distinguishes rank-based from Pearson correlation.
- 4.2 Results: ROUGE-WE-2 improves on ROUGE-2 most of the time across the correlation metrics, whereas ROUGE-WE-SU4 is less amenable to composition because skip-bigrams lose semantic meaning.The paper attributes the difference partly to multiplicative composition working better for contiguous bigrams than for skip-bigrams.
- 4.2 Results: ROUGE-WE-1 displays better correlations with pyramid scores than the top system in AESOP 2011 under the Spearman comparison.The comparison is made against other leading AESOP systems, including graph-based and baseline systems.
5 Conclusion
The paper enhances ROUGE with word embeddings, achieving strong agreement with human assessments while moving beyond surface lexical matching. It identifies dataset coverage and compositional embeddings as priorities for further improvement.
- ROUGE-WE incorporates word embeddings to capture semantic similarities between generated and human-written summaries, rather than relying only on lexical matches.
- Experiments on the TAC AESOP dataset show very good correlations with human assessments using Spearman and Kendall rank coefficients.
- The AESOP evaluation summaries are strongly biased toward extractive systems, motivating a more inclusive dataset containing substantially paraphrased outputs.
- More effective compositional embedding models remain an open direction, particularly for improving bigram ROUGE-WE variants such as ROUGE-WE-SU4.
- Word embeddings offer a promising dimension for developing more effective automatic summarization evaluation measures.