Source-linked AI summary
ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks
Kavita Ganesan
TL;DR
Existing ROUGE measures capture surface n-gram overlap but provide limited coverage of synonymous concepts and topics, complicating interpretation of summary quality. ROUGE 2.0 introduces synonym-aware and topic-focused measures, with examples showing that topic-based scores can better reflect captured opinions and that topic-unique scoring suppresses repetition. The paper also reports an implementation discrepancy for ROUGE-N scores when N > 1 between the Perl and Java packages.
Problem
Existing ROUGE measures may not reflect summary quality because they omit synonymous concepts and topic or subset coverage.
Method
ROUGE 2.0 extends ROUGE with synonym dictionaries, customizable topic evaluation, and unique topic or subset coverage measures in a Java implementation.
Results
ROUGE-TopicNN|JJ recall indicates that both example summaries capture most reference opinions, scores improve with synonyms, and ROUGE-TopicUniqNN|JJ further improves precision by suppressing repetition.
Takeaways & Limitations
Evaluating semantic overlap and selected topics can provide a more multifaceted view of summary content than surface n-gram overlap alone.
Takeaways & Limitations
Topic definitions depend on the application, and topic-based score improvements are not guaranteed in every case.
Abstract
from arXiv · showhide
Evaluation of summarization tasks is extremely crucial to determining the quality of machine generated summaries. Over the last decade, ROUGE has become the standard automatic evaluation measure for evaluating summarization tasks. While ROUGE has been shown to be effective in capturing n-gram overlap between system and human composed summaries, there are several limitations with the existing ROUGE measures in terms of capturing synonymous concepts and coverage of topics. Thus, often times ROUGE scores do not reflect the true quality of summaries and prevents multi-faceted evaluation of summaries (i.e. by topics, by overall content coverage and etc). In this paper, we introduce ROUGE 2.0, which has several updated measures of ROUGE: ROUGE-N+Synonyms, ROUGE-Topic, ROUGE-Topic+Synonyms, ROUGE-TopicUniq and ROUGE-TopicUniq+Synonyms; all of which are improvements over the core ROUGE measures.
1. Problems with the current ROUGE measures
The paper argues that conventional ROUGE can misrepresent summary quality because surface n-gram overlap misses synonymous concepts and topic or subset coverage. It proposes ROUGE 2.0 measures designed to address these limitations.
- ROUGE evaluates summaries by counting overlapping words or n-grams with human-created reference summaries and is primarily recall-oriented.
- ROUGE scores can understate quality when summaries express the same essential content with different words or levels of verbosity.The paper notes that a concise accurate summary or a verbose reference can make overlap appear artificially low.
- 0.727 versus 0.286: ROUGE-1 F-Score with stop words for SysSum1 and SysSum2, respectively, despite both summaries capturing the reference’s main points.Without stop words, the corresponding scores are 0.571 and 0.305.
- Synonymous concepts such as ‘display’ and ‘screen’ are treated as different words, producing lower scores than their semantic overlap would warrant.The paper identifies synonym capture as one way to reduce this problem.
- ROUGE also misses topic or subset coverage, although applications may require evaluating selected content such as nouns, adjectives, entities, or opinions.Without topic coverage information, systems may be pushed toward reproducing reference summaries rather than capturing essential content.
- ROUGE 2.0 proposes synonym-aware ROUGE-{N|Topic|TopicUniq}, ROUGE-Topic, and ROUGE-TopicUniq measures, with customizable synonym dictionaries and topic definitions.The topic definition can be customized by part-of-speech occurrence.
2. ROUGE 2.0
ROUGE 2.0 updates ROUGE with synonym-aware scoring and topic or subset coverage, implemented in a modular Java package. Its measures support configurable synonym dictionaries, POS-defined topics, and unique-topic matching to address limitations of surface n-gram overlap.
- ROUGE 2.0 is a Java implementation with updated scoring for semantic overlap and specific topic or subset evaluation.The package extends ROUGE through synonym dictionaries and topic-based measures.
- Synonym-aware scoring: ROUGE-{NN|Topic|TopicUniq}+Synonyms uses a customizable synonym dictionary so semantically similar words can count as one.The default English implementation uses WordNet for nouns, verbs, and adjectives, while domain- or language-specific dictionaries can replace it.
- Topic or subset coverage: ROUGE-Topic evaluates selected summary dimensions using configurable part-of-speech combinations, such as nouns for entities or nouns and adjectives for opinion coverage.It normally uses unigrams because topical word occurrence matters more than word order or co-occurrence.
- Unique-topic scoring: ROUGE-TopicUniq counts set intersections of unique topic tokens, suppressing the effect of repeated topic words on precision and recall.The measure uses unique unigram sets from reference and system summaries for the selected POS.
- Results: In Example 1.1, ROUGE-TopicNN|JJ recall indicates that both systems captured most reference opinions, while synonyms further improve these scores.ROUGE-TopicUniqNN|JJ improves precision agreement by counting topic matches once; SysSum1 recovers all opinions concisely, whereas SysSum2 retains lower precision than recall.
- Package implementation: The Java package is platform independent and simplifies summary naming and formatting, supporting broader use than the original Perl package.The implementation does not replicate the unexplained add-1 behavior observed in Perl ROUGE-N scores for N > 1.