Source-linked AI summary
Neural Text Summarization: A Critical Evaluation
Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, Richard Socher
TL;DR
Text summarization research has made substantial progress in models and datasets, but benchmark performance has stagnated amid weaknesses in the research setup. The paper critically evaluates datasets, evaluation metrics, and model outputs, finding underconstrained and noisy data, weak metric correlation with human judgment, and reliance on layout bias. It concludes that more robust datasets, less domain-biased models, and broader evaluation are needed within the supported scope.
Problem
Benchmark progress has stagnated, while automatically collected datasets, current evaluation protocols, and model outputs exhibit important shortcomings.
Method
The paper critically analyzes datasets, evaluation metrics, and model outputs, including dataset constraints and noise, human-metric correlations, and layout bias.
Results
The analysis finds that datasets can be underconstrained and noisy, evaluation correlates weakly with human judgment, and models rely heavily on layout bias.
Takeaways & Limitations
The paper calls for better-constrained datasets, models less fitted to particular domain biases, and evaluation beyond current metrics.
Takeaways & Limitations
News-domain layout heuristics may not apply to books or legal documents and should be tested through ablation studies rather than assumed by default.
Abstract
from arXiv · showhide
Text summarization aims at compressing long documents into a shorter form that conveys the most important parts of the original document. Despite increased interest in the community and notable research effort, progress on benchmark datasets has stagnated. We critically evaluate key ingredients of the current research setup: datasets, evaluation metrics, and models, and highlight three primary shortcomings: 1) automatically collected datasets leave the task underconstrained and may contain noise detrimental to training and evaluation, 2) current evaluation protocol is weakly correlated with human judgment and does not account for important characteristics such as factual correctness, 3) models overfit to layout biases of current datasets and offer limited diversity in their outputs.
1 Introduction
Text summarization compresses documents into concise, readable summaries through extractive or abstractive generation. Despite rapid growth in neural models and datasets, benchmark progress has stagnated, motivating scrutiny of the research setup.
- Extractive systems copy salient source fragments, whereas abstractive systems paraphrase salient content into the final summary.
- Neural summarization has expanded through advanced architectures, large-scale data, attention and copying mechanisms, reinforcement learning, and hybrid models.
- State-of-the-art models only slightly outperform Lead-3, suggesting that uncurated datasets and non-informative evaluation protocols contribute to benchmark stagnation.
2 Related Work
Related work spans large-scale datasets, automatic evaluation metrics, and extractive, abstractive, and hybrid models. The paper positions its critique within broader efforts to question established methods and research settings.
- Datasets: Large-scale summarization corpora predominantly come from the news domain, including Gigaword, NYT, CNN/DailyMail, and XSum.
- Datasets: Reddit TIFU and WikiHow extend dataset coverage beyond news to personal stories, procedural instructions, and other structured online content.
- Evaluation: Automatic metrics were developed because manual evaluation of large-scale systems is costly and cumbersome.
- Evaluation: ROUGE primarily measures exact lexical overlap between candidate and reference summaries, while later variants incorporate paraphrase matching or semantic similarity.
- Models: Existing models are categorized as abstractive, extractive, or hybrid, with hybrid systems separating content selection from paraphrasing.
- Critical evaluation: The paper follows related work that scrutinizes established methodologies, datasets, and models rather than focusing only on new architectures or training strategies.
3 Datasets
The paper examines how summarization datasets make content selection underconstrained, encode news-layout biases, and contain noisy examples. Human studies and corpus inspection show ambiguity in what counts as important, concentration of important information near article beginnings, and measurable dataset flaws.
- 3.1 Underconstrained task: Summarization importance depends on the target reader’s expectations and prior knowledge, making content selection difficult to specify.
- 3.1 Underconstrained task: With one reference summary and no additional information, the task is underconstrained and ambiguous for end-to-end models.
- 3.1 Underconstrained task: At a threshold of three human votes, annotators agreed on 0.627 unconstrained and 1.392 constrained important sentences per article.The corresponding sampled articles averaged 16.59 sentences, with a standard deviation of 5.39.
- 3.1 Underconstrained task: Constrained summaries were more succinct and targeted, while unconstrained summaries tended to be more verbose without adding information.Human summaries also expressed similar content in unique ways, highlighting their abstractive nature.
- 3.2 Layout bias in news data: Nearly 60% of important information appeared in the first third of CNN/DailyMail articles, versus approximately 25% in the second and 15% in the last third.The distribution of human-selected important sentences was skewed toward the article’s first quarter.
- 3.3 Noise in scraped datasets: Automatic scraping makes corpus quality depend heavily on whether scraped content matches authors’ structural assumptions, while simple heuristics remove only obviously flawed examples.Manual inspection found noisy reference summaries containing links, placeholders, unparsed HTML, and non-informative passages; noise affected 0.47%, 5.92%, and 4.19% of CNN/DM training, validation, and test splits, and 3.21%, 3.22%, and 3.17% of the corresponding Newsroom splits.
4 Evaluation Metrics
The paper finds that ROUGE correlates weakly with human judgments in the current CNN/DM evaluation setting and does not directly assess factual consistency. Manual inspection also found consistency issues in 30% of sampled abstractive summaries.
- ROUGE showed minimal Pearson correlation with annotated dimensions for abstractive models, while extractive models had weak correlation mainly for fluency and coherence.
- Model rankings showed weak correlation across dimensions for mixed and abstractive models, but moderate correlation for extractive models.
- Correlation weakened as the number of ground-truth references increased, contrary to the expectation that more references would improve agreement.
- ROUGE and common human evaluations do not explicitly examine factual consistency, despite factual alignment being a goal of summarization.
- 30% of 200 manually reviewed article-summary pairs contained consistency issues.
5 Models
The paper examines whether models rely on news-layout cues and whether architecturally different systems produce diverse outputs. Results indicate strong layout-bias effects and greater shared token-level than phrase-level overlap.
- Layout bias in news data: Current models were evaluated against both CNN/DM references and Lead-3 references to quantify reliance on leading sentences.Lead-3 uses the first three sentences of each source article as the reference.
- Layout bias in news data: All examined models showed substantially higher overlap when Lead-3 was used as the reference across ROUGE variants.
- Layout bias in news data: Lead-3 is a strong layout-bias baseline, although extractive-oracle performance remains substantially higher.
- Diversity of model outputs: Pairwise ROUGE-1 and ROUGE-4 compared model outputs through token and phrase overlap.
- Diversity of model outputs: ROUGE-1 scores varied considerably less than ROUGE-4 scores, indicating shared vocabulary but differing longer-phrase organization.
- Diversity of model outputs: Model-pair overlap exceeded model-reference overlap, suggesting shared patterns in training data or weak source-reference connections.
6 Conclusions
The conclusion identifies three weaknesses in the current summarization research setup: insufficient dataset constraints, excessive layout-bias reliance, and evaluation that weakly reflects human judgment while missing factual correctness.
- Datasets require additional constraints to produce well-formed summaries.
- State-of-the-art methods rely too heavily on layout bias associated with the domain of the source text.
- Current evaluation reflects human judgments only weakly and fails to evaluate critical features such as factual correctness.
- The paper proposes future research on datasets, models less fitted to domain bias, and evaluation beyond current metrics.
A Human study details
The human studies used CNN/DM test examples and balanced annotation quality against sample size and experimental breadth.
- Human studies sampled examples from the CNN/DM test split, which contains 11,700 examples.
- The study design traded off the number of annotated examples, experimental breadth, and annotation quality.
A.1 Underconstrained task
The study used human-written summaries and source highlights, while constrained-setting questions were prepared separately and curated before summary collection.
- Human annotators wrote summaries of news articles and highlighted source fragments useful for writing them.
- The study covered 100 randomly sampled articles, each annotated by 5 unique annotators in both settings.
- Constrained-setting questions were written by human annotators in a separate assignment and curated before collection.
A.2 ROUGE - Weak correlation with human judgment
The study evaluated 13 neural summarization models across randomized experiment groups, using a shared reference point to align comparisons between groups.
- 13 neural models were evaluated: 10 abstractive and 3 extractive.
- The experiment was split into 3 groups, with two groups containing 4 models and one containing 5 models.
- Models were assigned to groups per example to randomize the evaluation context and reduce biased data collection.
A.3 Layout bias in news data
Human annotators read news articles and highlighted sentences containing the most important information, using 100 articles with 5 annotators per article.
- Annotators read news articles and highlighted sentences containing the most important information.
- The study used 100 randomly sampled articles.
- Each article was annotated by 5 unique annotators.