Source-linked AI summary
Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Max Grusky, Mor Naaman, Yoav Artzi
TL;DR
Summarization research is constrained by limited high-quality data and diverse writing strategies. NEWSROOM contributes a large human-written dataset, measures its extractive–abstractive diversity, and evaluates existing systems. The dataset exposes an open challenge while supporting data-intensive learning and improving out-of-domain performance.
Problem
Limited high-quality data and diverse summarization strategies constrain training and evaluation of automatic summarization methods.
Method
NEWSROOM assembles human-written summaries from newsrooms, measures extractiveness and abstractiveness, and evaluates existing summarization models with automated and human protocols.
Results
NEWSROOM’s diverse summaries present an open challenge, while training an existing mixed-strategy model on the dataset improves performance on out-of-domain data.
Takeaways & Limitations
NEWSROOM provides a large resource for data-intensive learning and new directions for evaluating summarization difficulty and developing future models.
Takeaways & Limitations
ROUGE evaluation biases scoring against abstractive systems because it relies on shared n-gram frequencies, motivating human evaluation.
Abstract
from arXiv · showhide
We present NEWSROOM, a summarization dataset of 1.3 million articles and summaries written by authors and editors in newsrooms of 38 major news publications. Extracted from search and social media metadata between 1998 and 2017, these high-quality summaries demonstrate high diversity of summarization styles. In particular, the summaries combine abstractive and extractive strategies, borrowing words and phrases from articles at varying rates. We analyze the extraction strategies used in NEWSROOM summaries against other datasets to quantify the diversity and difficulty of our new data, and train existing methods on the data to evaluate its utility and challenges.
1 Introduction
NEWSROOM addresses limited high-quality summarization data with 1.3 million human-written article summaries from diverse publishers and practical newsroom settings. Its mixed extractive–abstractive styles create a challenging benchmark for summarization systems.
- 1.3 million news articles and human-written summaries form NEWSROOM, addressing limited high-quality data for summarization training and evaluation.
- Authors and editors produced the summaries in newsrooms across news, sports, entertainment, financial, and other publications.
- The summaries were written for common readers with the explicit purpose of summarization, providing a nearly two-decade snapshot of practical single-document summarization.
- NEWSROOM includes summaries from 38 major publishers, whose diverse sources and time span yield diverse summarization styles.
- The analysis measures extractiveness and abstractiveness to divide summaries into extractive, mixed, and abstractive subsets.
- NEWSROOM presents an open challenge for summarization systems while providing a large resource for data-intensive learning methods.
2 Existing Datasets
Existing summarization datasets offer useful scale or evaluation properties but often rely on simulated, specialized, small, or extraction-biased summaries. Their differing construction choices affect how well they represent general summarization and complicate comparisons.
- Existing datasets use headlines, concatenated bullet points, or archival summaries to obtain large-scale summarization data.
- 2.1 Document Understanding Conference: DUC provides small, high-quality newswire datasets with human-written summaries and multiple reference summaries for evaluation.
- 2.2 Gigaword: Gigaword contains nearly 10 million documents from seven newswire sources, but its headlines are used as simulated summaries of article first sentences.
- Existing datasets show style constraints: Gigaword summaries are shorter than natural summary text, while New York Times and CNN / Daily Mail summaries are biased toward extraction.
- 2.3 New York Times Corpus: The New York Times Annotated Corpus contains several hundred thousand paired articles and summaries from one source, with summaries written by library scientists after publication.
- CNN / Daily Mail pairs articles with bullet-point descriptions that are concatenated for summarization, making system comparisons challenging because preprocessing and training data differ.
3 Collecting NEWSROOM Summaries
NEWSROOM was built by crawling archived publisher pages, extracting article text and metadata summaries, and filtering and deduplicating the resulting article-summary pairs.
- The dataset was collected from archived pages of popular news, sports, and entertainment sites using Internet Archive search and crawling tools.Archive.org enabled collection across time and distribution through immutable archived URLs.
- Publisher selection combined Alexa overall and news rankings with older Google high-traffic lists, excluding aggregators, unsuitable sites, and primarily non-English publishers.
- Article URLs were deduplicated by retaining the earliest available version, preventing stale summaries from changed articles.
- Readability extracted article bodies, while cleaning excluded articles without summaries and pairs with excessive exact overlap from automatically copied summaries.
- 1,321,995 article-summary pairs resulted from the scraping and extraction process.
- The data were split into training, development, test, and unreleased test sets using a hash of each article URL.The proportions were 76%, 8%, 8%, and 8%, respectively.
4 Data Analysis
NEWSROOM characterizes summaries using extractive-fragment measures and compression, revealing substantial variation across publishers and a broader range of styles than existing datasets.
- Three measures quantify text overlap and information compression: extractive fragment coverage, density, and compression ratio.The analysis uses these measures to compare human summarization techniques across sources.
- Characterizing Summarization Strategies: Extractive fragments are identified greedily by matching each summary position to the longest possible source-text prefix.Unmatched summary tokens are treated as abstractive.
- Characterizing Summarization Strategies: Coverage measures the percentage of summary words belonging to extractive fragments from the article.A 10-word summary borrowing 7 words has coverage 0.7.
- Characterizing Summarization Strategies: Density measures the average extractive-fragment length assigned to each summary word, using squared fragment lengths in its formulation.
- Characterizing Summarization Strategies: Compression is defined as the word ratio between the article and summary, and higher compression requires more precise capture of critical article aspects.
- Analysis of Dataset Diversity: NEWSROOM spans a much larger range of summarization styles than existing datasets, from highly extractive to highly abstractive.CNN / Daily Mail and New York Times are skewed toward extractive summaries, while DUC has much higher median compression.
5 Performance of Existing Systems
The study evaluates extractive, abstractive, and mixed summarization systems on NEWSROOM, using automated ROUGE and human judgments to assess performance and challenges.
- Three systems represent distinct extraction strategies: fully extractive TextRank, fully abstractive Seq2Seq, and mixed pointer-generator models.
- TextRank selects sentences using an unsupervised sentence-ranking approach and is tuned to a 50-word maximum for NEWSROOM ROUGE performance.
- Seq2Seq with attention generates tokens without an explicit mechanism for copying from the input text.
6 Automatic Evaluation
The paper evaluates summarization systems with ROUGE across DUC, CNN/Daily Mail, and NEWSROOM, including NEWSROOM subsets grouped by extractiveness. NEWSROOM-trained pointer models generally perform strongly, especially on out-of-domain data and extractive subsets.
- The evaluation uses F1 variants of ROUGE-1, ROUGE-2, and ROUGE-L to account for different summary lengths.
- Pointer-N and Pointer-S outperform other systems and baselines on DUC, while Pointer-C does not outperform Lede-3.
- Among models, NEWSROOM-trained Pointer-N performs best on every dataset except CNN / Daily Mail.
- Pointer-N outperforms the remaining models across all extractive NEWSROOM subsets and exceeds Lede-3 on the abstractive subset.
7 Human Evaluation
The paper supplements ROUGE with human evaluation because n-gram overlap can bias scoring against abstractive summaries. The protocol separates semantic and syntactic quality while aiming for reliable, portable measurement.
- Human evaluation addresses ROUGE’s bias against abstractive systems that rely more on paraphrasing.The authors frame NEWSROOM as an opportunity to develop a benchmark-specific human-evaluation protocol.
- The evaluation targets semantic and syntactic summarization quality, reliable measurement, and portability across models or datasets.Its objectives include distinguishing quality dimensions, ensuring consistency and replicability, and enabling reuse.
- The four dimensions are informativeness, relevance, fluency, and coherence.Informativeness and relevance assess semantic usefulness and source appropriateness, while fluency and coherence assess writing quality.
- TextRank received the highest overall human-evaluation score, with raters favoring its fluency and coherence.No system exceeded the Lede-3 baseline, while the extractive oracle underperformed most systems and Abs-N improved slightly relative to its automatic evaluation.
8 Conclusion
The paper introduces NEWSROOM as a large, diverse summarization dataset and uses its diversity analysis to motivate new evaluation and modeling directions.
- NEWSROOM contains articles and summaries written in online publication newsrooms and is presented as the largest summarization dataset available at the time.
- The dataset exhibits a wide variety of human summarization strategies.
- The proposed measures and strategy analyses suggest directions for evaluating summarization difficulty and developing future models.
Additional Evaluation
Additional evaluation measures summarization diversity and examines how system performance varies across reference summaries with different extractive strategies and compression levels.
- Coverage, density, and compression quantify summarization diversity and support system error analysis.
- Coverage: The coverage subsets compare systems on references ranging from more novel wording at low coverage to greater article-word borrowing at high coverage.Table 6 defines the subsets by how frequently references borrow words from their source texts.
- Compression: The compression subsets compare longer low-compression references with shorter high-compression references relative to their articles.Table 7 defines compression by reference-summary length relative to the source text.