Source-linked AI summary
A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal
Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, Georgiana Ifrim
TL;DR
Large-scale MDS datasets that reflect real-world news collections are scarce, limiting deep-model use and robust evaluation. This paper presents WCEP, a large dataset of concise event summaries and news articles; its experiments find that additional articles raise extractive upper bounds and help improve baseline MDS methods.
Problem
MDS lacks large-scale datasets that realistically represent news-event collections, limiting deep-model use and making robust evaluation difficult.
Source: §1 · p. 1Method
Results
Additional articles yield much higher upper bounds for standard extractive summarization and help improve baseline MDS methods.
Source: §1 · p. 2Takeaways & Limitations
WCEP is intended to facilitate real-world MDS systems for summarizing news clusters or search results.
Source: §6 · p. 5
Abstract
from arXiv · showhide
Multi-document summarization (MDS) aims to compress the content in large document collections into short summaries and has important applications in story clustering for newsfeeds, presentation of search results, and timeline generation. However, there is a lack of datasets that realistically address such use cases at a scale large enough for training supervised models for this task. This work presents a new dataset for MDS that is large both in the total number of document clusters and in the size of individual clusters. We build this dataset by leveraging the Wikipedia Current Events Portal (WCEP), which provides concise and neutral human-written summaries of news events, with links to external source articles. We also automatically extend these source articles by looking for related articles in the Common Crawl archive. We provide a quantitative analysis of the dataset and empirical results for several state-of-the-art MDS techniques.
1 Introduction
Large-scale MDS datasets are scarce despite their importance for real-world collections of news articles, limiting deep-model use and robust evaluation. This paper introduces WCEP, a large dataset of concise event summaries and news articles, and reports that added articles improve extractive upper bounds and baseline MDS performance.
1 Introduction
- MDS lacks established large-scale datasets, impeding deep-learning model use and making robust evaluation difficult.The section motivates better datasets for news-event applications including event detection, article search, and timeline generation.Source: §1 · p. 1
- WCEP provides 10,200 clusters, each with one human-written summary and an average of 235 articles, targeting real-world MDS use cases.The dataset combines WCEP source articles with similar articles found in Common Crawl, creating large, highly redundant clusters.Source: §1 · p. 1
- Added articles yield much higher upper bounds for standard extractive summarization and help improve baseline MDS methods.The experiments test unsupervised and supervised MDS methods.Source: §1 · p. 2
- The paper presents WCEP as a large-scale MDS dataset better aligned with real-world industrial use cases.Source: §1 · p. 2
- The authors provide dataset-property analysis and empirical results for baseline and state-of-the-art MDS methods to support future work.Source: §1 · p. 2
2 Related Work
Prior MDS methods include extractive sentence ranking or global optimization, as well as abstractive compression, fusion, and neural approaches. Existing datasets differ in scale, source selection, and summary style; WCEP is characterized by short news-event summaries and large article clusters.
2.1 Multi-Document Summarization
- Extractive MDS commonly ranks sentences by importance or uses global optimization to select combinations, while abstractive work includes compression, fusion, and neural sequence-to-sequence approaches.Neural approaches have been applied to extractive summaries or directly to multiple documents.Source: §2.1 · p. 2
2.2 Datasets for MDS
- Traditional DUC 2004 and TAC 2011 datasets contain only 50 and 100 clusters, respectively, with 10 articles per cluster on average; MultiNews has 56,000 clusters but averages 2.3 sources.MultiNews sources were hand-picked and its summaries are more verbose than WCEP's.Source: §2.2 · p. 2
- Auto-hMDS and WikiSum use Wikipedia-based summaries and web-retrieved sources, but have long summaries on varied topics rather than WCEP's short news-event focus.Auto-hMDS has 7,300 clusters; WikiSum contains 2.3 million clusters.Source: §2.2 · p. 2
3 Dataset Construction
The dataset pairs concise WCEP event summaries with archived linked news articles, then expands clusters using Common Crawl articles classified as relevant to each event. It also provides chronological splits and a truncated version for experiments.
Wikipedia Current Events Portal
- WCEP summaries are short, complete present-tense sentences that avoid opinions and sensationalism, and events must be of international interest.The guidelines specify summaries up to 30–40 words; sources and summaries are preferably in English.Source: §3 · p. 2
Obtaining Articles Linked on WCEP
- The original WCEP source articles are snapshotted through the Internet Archive and extracted from the Wayback Machine.The process is intended to prevent source articles from becoming unavailable over time.Source: §3 · p. 2
Additional Source Articles
- WCEP events average 1.2 linked sources, motivating the search for similar articles in Common Crawl News.Source: §3 · pp. 2–3
- Additional Common Crawl articles are selected by a logistic-regression article-summary classifier trained on WCEP source pairs, with same-date cross-event negatives at a 7:100 positive-negative ratio.The classifier uses TF-IDF-weighted unigram vectors and cosine similarity for its first two features, and exact matching of linked entities in article titles and bodies; hold-out precision is 90% and recall is 74%.Source: §3 · p. 3
- Articles published within ±1 day of an event are added when classifier probability exceeds 0.9, and articles assigned to multiple events go to the highest-probability event.This procedure considerably increases the number of articles per summary.Source: §3 · p. 3
Final Dataset
- The 10,200-cluster dataset is split chronologically into roughly 80% training, 10% validation, and 10% test, with no event dates overlapping across splits.The truncated WCEP-100 version retains original articles and randomly samples additional articles up to 100 per cluster.Source: §3 · p. 3
4 Dataset Statistics and Analysis
The dataset analysis describes cluster-size statistics, evaluates the relevance of Common Crawl additions, and examines how WCEP summaries overlap with source articles. Added articles raise coverage, while high-density copying is less common in WCEP than in MultiNews.
4.1 Overview
- The overview reports partition sizes and individual-cluster statistics for WCEP-total and the experiment version WCEP-100; Common Crawl articles account for much of the high mean cluster size.WCEP-100 is the truncated dataset version referenced in the analysis.Source: §4.1 · p. 3, Table 3 · p. 3, Table 4 · p. 3
4.2 Quality of Additional Articles
- Manual review classified 52% of 350 additional articles as on-topic, 30% as related, and 18% as unrelated.The authors consider this noise acceptable because it resembles noise in automatic content aggregation; their experiments also find summarization benefits from added articles.Source: §4.2 · p. 3
4.3 Extractive Strategies
- WCEP coverage increases as Common Crawl articles are added, indicating that summary words tend to appear in larger clusters.High coverage suggests retrieval and copy mechanisms within a cluster can be useful for generating summaries.Source: §4.3 · p. 4, Figure 1 · p. 4
- WCEP summaries have less high-density copying than MultiNews, likely reflecting their short style and editor guidelines.Coverage measures the proportion of summary words in shared fragments; density reflects their average length and how well summaries can be described as extractions.Source: §4.3 · p. 4
5 Experiments
Experiments compare extractive and abstractive baselines on WCEP-100, capped at 100 articles per cluster for scalability. Supervised methods modestly outperform unsupervised ones, and added Common Crawl articles improve performance, especially at small additions.
5.1 Setup
- WCEP-100 caps clusters at 100 articles because some methods have scalability issues; summaries are limited to 40 tokens.The methods considered begin to plateau after 100 articles per cluster.Source: §5.1 · p. 4
5.2 Methods
- The study evaluates ROUGE-1, ROUGE-2, and ROUGE-L F-score and Recall, alongside unsupervised and supervised methods.Unsupervised methods include TextRank, Centroid, and Submodular; supervised methods include TSR and BERTREG, with oracle baselines used to contextualize scores.Source: §5.1 · p. 4, §5.2 · pp. 4–5
5.3 Results
- Supervised TSR and BERTREG outperform unsupervised methods on WCEP, but only by a small margin.The high ORACLE (SINGLE) extractive bound suggests identifying important documents before summarization may be useful; lead summaries perform poorly in F-scores, and SUBMODULAR + ABS does not perform well.Source: §5.3 · p. 5
5.4 Effect of Additional Articles
- Adding 10 Common Crawl articles causes a steep performance improvement over using only WCEP source articles, while additions beyond 100 yield minimal gains.The methods' performance starts to plateau after 100 articles per cluster.Source: §5.1 · p. 4, §5.4 · p. 5
6 Conclusion
The paper presents a news-domain MDS dataset of large article clusters paired with short event summaries, intended for real-world uses such as summarizing news clusters or search results. It provides baseline results and identifies scaling deep-learning methods and closing the gap to oracle methods as future challenges.
6 Conclusion
- The dataset pairs large news-article clusters with short event summaries, supporting uses such as summarizing news clusters or search results.The authors hope it will facilitate real-world MDS systems.Source: §6 · p. 5
- The authors hope future MDS work will use the dataset as a benchmark, with challenges including scaling deep learning to large source collections and closing the gap to oracle methods.Source: §6 · p. 5
A.1 BERTREG
BERTREG scores sentences with a regression model and uses the scores to construct summaries. Its training target is the sentence's ROUGE-1 F-score against the cluster summary.
A.1 BERTREG
- Table 6 lists the features used for BERTREG apart from its contextual sentence embeddings.Source: Table 6 · p. 7
- The model predicts a sentence's ROUGE-1 F-score against the cluster summary using a feedforward network with one hidden layer of size 256.It is trained with mean squared error; the authors report that F-score worked better than Precision or Recall.Source: §A.1 · p. 7
- At summary time, BERTREG ranks sentences by predicted score and greedily selects them under a redundancy constraint, as in TSR.Source: §A.1 · p. 7
A.2 Implementation Details for Extractive Methods
The extractive methods use a shared greedy sentence-selection framework that avoids redundancy. The implementation tunes selection parameters on validation clusters and uses a minimum sentence length to avoid very short or broken sentences.
A.2 Implementation Details for Extractive Methods
- TEXTRANK, CENTROID, TSR, and BERTREG use a shared framework that greedily selects ranked sentences while avoiding redundancy.Source: §A.2 · p. 7
- Redundancy thresholds are tuned from 0 to 1 in steps of 0.1, while SUBMODULAR diversity is tuned from 1 to 10 in steps of 1.Tuning uses 100 randomly selected WCEP-100 validation clusters.Source: §A.2 · p. 7
- A minimum sentence length of 7 tokens is used to avoid very short or broken sentences in summaries near the 40-token limit.Source: §A.2 · p. 7
- BERTREG uses the 12-layer model from the cited BERT service implementation.Source: §A.2 · p. 7