Source-linked AI summary
SummEval: Re-evaluating Summarization Evaluation
Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, Dragomir Radev
TL;DR
Summarization evaluation lacks comprehensive, current evidence and consistent protocols, limiting reliable comparison of metrics and models. SummEval addresses this by consistently re-evaluating metrics and models while releasing aligned outputs, a unified toolkit, and human judgments. Its analyses show that expert and crowd annotations can diverge, and that metric correlations vary across quality dimensions.
Problem
Current summarization evaluation uses inconsistent protocols, limited comparisons, and metric validations based on older DUC and TAC judgments that may not match contemporary systems.
Method
SummEval re-evaluates 14 automatic metrics and benchmarks 23 models using neural outputs, expert and crowd-sourced judgments, shared outputs, and a unified toolkit.
Results
Expert and crowd-sourced annotations showed no correlation, while metric correlations varied across quality dimensions and models such as Pegasus, BART, and T5 performed best on most human-evaluation dimensions.
Takeaways & Limitations
SummEval provides a broad, consistent resource for comparing summarization models and developing evaluation metrics that better align with human judgments.
Abstract
from arXiv · showhide
The scarcity of comprehensive up-to-date studies on evaluation metrics for text summarization and the lack of consensus regarding evaluation protocols continue to inhibit progress. We address the existing shortcomings of summarization evaluation methods along five dimensions: 1) we re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion using neural summarization model outputs along with expert and crowd-sourced human annotations, 2) we consistently benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics, 3) we assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset and share it in a unified format, 4) we implement and share a toolkit that provides an extensible and unified API for evaluating summarization models across a broad range of automatic metrics, 5) we assemble and share the largest and most diverse, in terms of model types, collection of human judgments of model-generated summaries on the CNN/Daily Mail dataset annotated by both expert judges and crowd-source workers. We hope that this work will help promote a more complete evaluation protocol for text summarization as well as advance research in developing evaluation metrics that better correlate with human judgments.
1 Introduction
Summarization evaluation has become standardized around CNN/DailyMail, ROUGE, and crowd-sourced judgments, but protocols and metric validation remain inconsistent. SummEval addresses these gaps through broad, consistent re-evaluation and shared resources.
- CNN/DailyMail pairs news articles with human bullet-point summaries, while ROUGE measures lexical overlap between generated and reference summaries.
- Recent studies report substantially different evaluation protocols, few baselines, and inconsistent human evaluations across summarization papers.
- Evaluation metrics themselves may be unreliable in current settings because DUC and TAC judgments score older model outputs lower than contemporary systems.
- SummEval re-evaluates 14 metrics and benchmarks 23 recent models using neural outputs plus expert and crowd-sourced human annotations.
- The project releases aligned outputs, a unified extensible toolkit, and human judgments to support large-scale comparisons and metric research.
2 Related Work
Prior summarization studies separately examined metrics, datasets, and models, revealing limitations in evaluation, data standardization, content selection, and factual faithfulness. SummEval complements this work with a consistent side-by-side analysis and shared resources.
- Earlier metric studies found that multiple references can improve correlation with human judgments, while annotation inconsistency destabilizes summary-level rankings.
- Dataset research called for unified standards and identified layout bias, underconstrained summarization targets, and noisy data in news corpora.
- Model analyses reported that extractive systems can reach abstraction levels similar to abstractive systems and that dataset biases impair content selection.
- Faithfulness studies found hallucinated facts in up to 30% of generated summaries and judged ROUGE and BertScore insufficient for studying factuality.
- SummEval provides the first large-scale, consistent, side-by-side re-evaluation of neural summarization outputs and evaluation methods, alongside reusable resources.
3 Evaluation Metrics and Summarization Models
SummEval evaluates diverse reference-based, reference-free, and structural metrics alongside extractive and abstractive summarization models. Its toolkit unifies these methods and makes model outputs easier to compare.
- Evaluation Metrics: The toolkit includes recently introduced text-generation metrics, machine-translation metrics, and miscellaneous performance statistics.
- Evaluation Metrics: ROUGE measures n-gram overlap, ROUGE-WE adds embedding-based soft lexical matching, S3 predicts scores from metric features, and BertScore greedily aligns contextualized tokens.
- Evaluation Metrics: MoverScore measures semantic distance through Word Mover’s Distance over BERT-derived n-gram embeddings, while SMS extends this idea to sentence embeddings.
- Evaluation Metrics: SummaQA uses question answering over masked source entities, BLANC is reference-free, and SUPERT compares outputs with pseudo-references.
- Summarization Models: Extractive and abstractive models are categorized separately and assigned model codes and descriptive names for matching against released data.
- Summarization Models: Extractive systems select or compress source sentences, whereas abstractive systems generate text using pointer, reinforcement-learning, pretrained, or denoising architectures.
4 Resources
The paper releases unified model outputs, an evaluation toolkit, and human annotations to support comparable summarization evaluation.
- Model outputs: The released outputs cannot be correctly aligned for 0.3% of the test data because duplicate reference summaries prevent reliable pairing.IDs of duplicate examples are provided with the data.
- Evaluation toolkit: The toolkit unifies 14 automatic metrics through common example-level and corpus-level evaluation functions.Its shared inputs and outputs streamline multimetric evaluation and result processing.
- Human annotations: Human evaluations cover 16 neural models rated by five crowd-sourced and three expert workers across four dimensions.The collection includes 12,800 summary-level annotations from 100 CNN/DailyMail articles.
- Human annotations: Judges rated grouped summaries from 1 to 5 using the source article and reference summary as common points of reference.Summary grouping and order were randomized for each annotator.
5 Metric Re-evaluation
The study re-evaluates human annotation quality and metric–human agreement, finding substantial disagreement among crowd workers and weak coverage of coherence and relevance by many metrics.
- Human annotations: 0.4920 and 0.4132 were the inter-annotator kappa values for crowd workers and first-round experts, versus 0.7127 after a second expert round.The second expert round lowered score dispersion and substantially improved agreement.
- Human annotations: Expert and crowd-sourced judges showed no correlation, with the statistic close to 0.Manual examples also revealed different interpretations of coherence, consistency, relevance, and fluency.
- Human annotations: Crowd-sourced ratings tended to be similar across quality dimensions even when expert annotations distinguished them.Examples included missed ambiguity, factual inconsistency, and repetition errors.
- Automatic metrics: Most metrics had weak or moderate correlation with coherence, while correlations increased for consistency and fluency.The authors relate coherence difficulty to metrics’ limited modeling of interdependence between consecutive sentences and note subjectivity in relevance.
- Automatic metrics: Metrics using higher-order n-gram overlap, including ROUGE-3 and Extractive Density, correlated more strongly with consistency than expected.The authors attribute this pattern to the low abstractiveness of most neural models and high expert agreement on consistency.
- Automatic metrics: Lexical-overlap metrics correlated strongly with one another, whereas novelty and repetitiveness metrics showed weak negative correlations with ROUGE-related metrics.Reference-less SummaQA, BLANC, and SuPERT also correlated weakly with most other metrics.
6 Model Re-evaluation
The model re-evaluation compares recent abstractive, extractive, and hybrid systems using expert ratings and automatic metrics, revealing strong performance from pretrained models but metric-specific preferences for extractive systems.
- Human evaluation: Pegasus, BART, and T5 consistently performed best on most human-evaluation dimensions.They scored highest on consistency and fluency but lower on relevance and coherence.
- Human evaluation: Extractive models showed weaker coherence and content-selection relevance in human ratings.Abstractive model ratings increased with publication date.
- Human evaluation: Reference summaries received low ratings for consistency, coherence, and relevance because they often included extraneous hyperlinks and click-bait descriptions.Annotators interpreted information absent from the source or irrelevant to the summary as hallucination.
- Automatic evaluation: Large models pretrained on vast quantities of data received the highest scores for most automatic metrics.These results aligned with the human-evaluation findings.
- Automatic evaluation: S3, SummaQA, SMS, CHRF, and METEOR tended to assign their highest scores to extractive outputs.This metric-specific preference differed from the broader pattern favoring large pretrained models.
7 Conclusions
SummEval combines shared resources with a consistent re-evaluation of summarization models and metrics. The authors position these resources as support for future research on summarization evaluation and models.
- SummEval provides model-generated CNN/DailyMail summaries, a unified evaluation toolkit, and diverse human annotations from crowd-source and expert judges.These resources support evaluation research on both summarization models and metrics.
- The authors re-evaluated a broad selection of current models and evaluation metrics using the accumulated resources in a consistent and comprehensive manner.
- The work encourages future contributions of model outputs and extensions to the evaluation toolkit with new metrics.
9 Appendix
The appendix describes the interface and process used to collect summary evaluations from crowd-source and expert annotators. Judges read the source article before evaluating associated summaries.
- Figure 3 presents the data collection interface used by both crowd-source and expert annotators.
- Annotators first read the source article and then evaluated associated summaries along relevance, consistency, fluency, and coherence.