Source-linked AI summary
Re-evaluating Evaluation in Text Summarization
Manik Bhandari, Pranav Gour, Atabak Ashfaq, Pengfei Liu, Graham Neubig
TL;DR
Automatic metrics are widely used because manual summarization evaluation is costly, yet it is unclear whether conclusions from older meta-evaluation datasets apply to modern systems. The paper builds a benchmark spanning top-scoring extractive and abstractive systems, multiple metrics, datasets, and evaluation levels, finding that metric conclusions differ across settings and datasets. These results motivate broader, regularly updated meta-evaluation.
Problem
It is unclear whether conclusions about automatic summarization metrics from older TAC datasets hold for modern systems and tasks.
Method
The paper creates a benchmark with top-scoring extractive and abstractive outputs, multiple automatic metrics, manual judgments, datasets, and system-level and summary-level evaluations.
Results
Metric behavior differs across datasets and settings: MoverScore is best on TAC but significantly worse than ROUGE-2 on CNN/DailyMail, while most metrics better compare extractive summaries than systems there.
Takeaways & Limitations
Meta-evaluation should span multiple datasets and application scenarios and be updated as summarization systems and datasets evolve.
Takeaways & Limitations
The top-k experiment does not perform significance testing because it contains few data points.
Abstract
from arXiv · showhide
Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -- for nearly 20 years ROUGE has been the standard evaluation in most summarization papers. In this paper, we make an attempt to re-evaluate the evaluation method for text summarization: assessing the reliability of automatic metrics using top-scoring system outputs, both abstractive and extractive, on recently popular datasets for both system-level and summary-level evaluation settings. We find that conclusions about evaluation metrics on older datasets do not necessarily hold on modern datasets and systems.
1 Introduction
Manual evaluation remains the gold standard, but its cost has driven widespread reliance on automatic metrics, especially ROUGE. This paper introduces a modern benchmark and finds that metric behavior and evaluation conclusions can differ substantially between older and newer datasets and systems.
- Motivation: Manual evaluation is the gold standard, but annotation time and cost have made automatic metrics the dominant practice in summarization research.ROUGE is by far the most popular automatic metric, despite limited discussion of how it may diverge from human judgments.
- Motivation: Classic TAC meta-evaluation datasets are 6–12 years old, so their conclusions may not hold for modern systems and summarization tasks.Earlier work found lower-quality TAC human summaries than modern systems and strong metric disagreement in the higher-scoring range.
- Benchmark: The benchmark contains outputs from 25 top-scoring extractive and abstractive systems on CNN/DailyMail, automatic evaluations from traditional and semantic metrics, and lightweight Pyramid judgments.The manual evaluations are used as a gold standard for evaluating both systems and automated metrics.
- Findings: The experiments show that metric properties observed on TAC can follow different trends on the newly collected CNN/DailyMail dataset.For example, MoverScore performs best on TAC but is significantly worse than ROUGE-2 on CNN/DailyMail.
- Future Research: The authors call for meta-evaluation across multiple datasets and application scenarios, regularly updated with evolving systems and datasets.They also suggest a shared task in which systems and metrics co-evolve.
2 Preliminaries
The paper compares summarization datasets, systems, metrics, and meta-evaluation procedures using summary-level and system-level correlations. These setups differ in whether correlations are computed among outputs for each document or across systems using mean human scores.
- Datasets: TAC-2008 and TAC-2009 are multi-document, multi-reference datasets, while CNN/DailyMail contains news articles paired with highlights.The CNN/DailyMail version used here does not anonymize entities.
- Systems: The study uses representative top-scoring CNN/DailyMail systems, including extractive and abstractive systems, with 14 abstractive outputs per document.The systems either achieve state-of-the-art or competitive performance and have publicly gatherable outputs.
- Metrics: Eight metrics compare system and reference summaries using lexical overlap, contextual embeddings, text distances, or distributional divergence.ROUGE-1, ROUGE-2, and ROUGE-L measure unigram, bigram, and longest-common-subsequence overlap; recall variants are used except for MoverScore.
- Correlation Measures: Pearson correlation measures linear association between two variables and is used for system-level meta-evaluation.Williams’ significance test addresses differences between correlations for dependent variables because metrics evaluated on one dataset are not independent.
- Summary Level: Summary-level correlation is computed within each document across its system outputs, then averaged across documents.For each document, metric scores are correlated with human scores among the different system summaries.
- System Level: System-level correlation compares metrics with human judgments across systems, whose quality is defined by each system’s mean human score.This setting reflects how automatic metrics are commonly used to rank systems against existing state-of-the-art systems.
3 Collection of Human Judgments
The study builds a human-evaluation benchmark for CNNDM by collecting outputs from 25 systems, sampling documents across difficulty levels, and adapting LitePyramids for single-reference summaries.
- Human-evaluation workflow: Human judgments follow three steps: collect CNNDM outputs, select representative test samples, and manually evaluate summaries for those samples.
- System collection: The authors collect outputs from 25 CNNDM systems, comprising 11 extractive and 14 abstractive systems.
- Document sampling: They sample 100 of 11,490 CNNDM test documents by partitioning documents into five difficulty bins and selecting four from each bin.Difficulty is estimated from the mean score received by system-generated summaries for each document.
- Single-reference adaptation: The study adapts LitePyramids because CNNDM has one reference summary, whereas Pyramid and LitePyramids normally use multiple references to weight Semantic Content Units.
- Annotation procedure: Each summary is evaluated by four crowd workers who mark sampled Semantic Content Units as present or not present.The example annotation uses reference-summary SCUs and records whether each can be inferred from a system summary.
- Annotation procedure: The dataset contains 10,000 annotations, after which noisy-worker filtering retains 7,742 annotations.
4 Experiments
The experiments evaluate metric reliability across system-level, top-k, pairwise-system, and summary-level settings. Results show that metric behavior varies substantially across datasets and evaluation levels, challenging conclusions drawn from older TAC benchmarks.
- 4.1 Exp-I: Evaluating All Systems: Metrics that perform well on TAC do not consistently correlate with human judgments on modern CNNDM systems, while ROUGE metrics perform consistently well on collected CNNDM datasets.MoverScore and JS-2 perform poorly on CNNDM extractive systems despite strong TAC performance; many metrics also degrade on TAC-2009, especially ROUGE-based metrics.
- 4.1 Exp-I: Evaluating All Systems: R-2 significantly outperforms other metrics on CNNDM abstractive and mixed systems, whereas no metric significantly outperforms others on CNNDM extractive systems.MoverScore and JS-2 are significantly better than other metrics on TAC datasets.
- 4.2 Exp-II: Evaluating Top-k Systems: As k decreases, metric correlations decline on TAC-2008 and CNNDM Mix, sometimes becoming negative, while several ROUGE metrics improve on CNNDM Ext.R-2 is negatively correlated on TAC-2009 for k < 50 but remains highly correlated across all k values on CNNDM Abs.
- 4.2 Exp-II: Evaluating Top-k Systems: Top-k evaluation shows that metrics cannot reliably quantify improvements among the strongest systems across all datasets.JS-2 and R-2 are identified as reliable indicators for TAC-2009 and CNNDM Abs, respectively.
- 4.3 Exp-III: Comparing Two-Systems: Pairwise system comparison finds ROUGE-based metrics moderately effective overall, with R-2 best on CNNDM and JS-2 highest on TAC-2009 but weak on CNNDM Ext.The study uses paired bootstrap resampling and weighted macro F1 to compare metric predictions with human-based pair labels.
- 4.4 Exp-IV: Evaluating Summaries: Summary-level and system-level correlations can diverge sharply: MoverScore rises from 0.05 system-level correlation to 0.74 summary-level correlation on CNNDM Ext.ROUGE metrics are weaker on TAC summary-level evaluation but strong indicators for extractive CNNDM summaries; some metrics generalize poorly across datasets.
5 Related Work
Related work situates the paper within human-judgment collection and automatic-metric meta-evaluation. Pyramid remains a mainstream method, while metric effectiveness is commonly assessed through correlation with human judgments and significance testing.
- Human Judgment Collection: Pyramid is a mainstream method for meta-evaluating automatic summarization metrics through human judgments of content selection.The method obtains semantic content units to support human evaluation.
- Meta-evaluation with Human Judgment: Automatic metrics including ROUGE-2, ROUGE-L, ROUGE-WE, JS-2, and S3 are commonly evaluated by correlating their scores with human judgments on TAC datasets.William’s test is identified as an important supplementary technique for assessing differences between metric correlations.
6 Implications and Future Directions
The paper argues that metric evaluation should keep pace with changing summarization datasets and systems, and identifies multidataset evaluation as a path toward more reliable metric selection.
- Metric choice depends on task, dataset, and application scenario, including whether evaluation is system-level or summary-level.
- Metrics can overfit limited datasets, so multidataset meta-evaluation can reveal each metric’s peculiarities across diverse settings.
- Human judgments from the benchmark could supervise a pretrain-then-finetune approach for learning a robust summarization metric.
A.1 Sampling Methodology
The sampling methodology constructs a sampled document set by randomly selecting documents, with Algorithm 1 defining the procedure and its output.
- Algorithm 1 is titled “Sampling Methodology” and specifies a procedure for constructing a sampled document set.
- The algorithm’s output is D_out, the sampled set of documents.
- The procedure randomly samples source documents d_i from D′′.
A.2 Exp-I using Kendall’s tau correlation
Experiment I evaluates system-level agreement between automatic metrics and human judgments using Kendall correlation across datasets.
- The experiment uses system-level Kendall’s tau correlation to compare different metrics with human judgments.
- Figure 7 presents Kendall correlation between metrics and human judgments across different datasets.
A.3 Exp-II using Kendall’s tau correlation
Experiment II examines Kendall correlation between metrics and human judgments at the summary level and for top-k systems.
- The experiment reports system-level Kendall’s tau correlation on top-k systems between metrics and human judgments.
- Figure 7 presents summary-level Kendall’s tau correlation between different metrics and human judgments.
- Figure 8 focuses on system-level Kendall correlation with human judgments among top-k systems.