Source-linked AI summary
Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model
Alexander R. Fabbri, Irene Li, Tianwei She, Suyi Li, Dragomir R. Radev
TL;DR
News MDS lacks the large datasets that have supported progress in SDS, despite the need to combine information across overlapping and differently focused articles. The paper introduces Multi-News and a hierarchical pointer-generator/MMR model, finding competitive performance across MDS benchmarks and improved non-redundancy in human evaluation. It also identifies limited attention to classical non-neural baselines as a scope boundary.
Problem
News MDS has been constrained by sparse, costly human-written datasets, even though summaries must comprehensively cover overlapping and divergent articles while avoiding redundancy.
Method
The paper creates the Multi-News dataset and integrates sentence-level MMR relevance and redundancy scores into a pointer-generator network for abstractive MDS.
Results
The model performs competitively on Multi-News and DUC 2004, while human evaluation finds Hi-MAP comparable to PG-MMR in informativeness and fluency but better in non-redundancy.
Takeaways & Limitations
Multi-News provides a large-scale news benchmark intended to support progress in multi-document summarization similar to advances in single-document summarization.
Takeaways & Limitations
The experiments focus on several non-neural baselines and leave broader classical-method implementation on Multi-News for future work.
Abstract
from arXiv · showhide
Automatic generation of summaries from multiple news articles is a valuable tool as the number of online publications grows rapidly. Single document summarization (SDS) systems have benefited from advances in neural encoder-decoder model thanks to the availability of large datasets. However, multi-document summarization (MDS) of news articles has been limited to datasets of a couple of hundred examples. In this paper, we introduce Multi-News, the first large-scale MDS news dataset. Additionally, we propose an end-to-end model which incorporates a traditional extractive summarization model with a standard SDS model and achieves competitive results on MDS datasets. We benchmark several methods on Multi-News and release our data and code in hope that this work will promote advances in summarization in the multi-document setting.
1 Introduction
Neural summarization has advanced mainly for single documents because large news datasets are available, while news-focused multi-document summarization remains data-sparse and challenging. Multi-News addresses this gap with a large-scale dataset, a hierarchical abstractive model, and benchmark evaluations.
- Research gap: News-focused multi-document summarization has relied on datasets with fewer than 100 document clusters, limiting the benefits of deep learning.Single-document datasets contain hundreds of thousands to millions of article-summary pairs, whereas MDS datasets such as DUC 2004 and TAC 2011 are much smaller.
- Research gap: MDS must organize overlapping and divergent reporting into comprehensive summaries while avoiding redundancy.Documents about one event may differ in focus and point of view, requiring summaries to combine shared and source-specific information.
- Contributions: Multi-News is the first large-scale news MDS dataset, containing 56,216 article-summary pairs.The dataset is introduced to address the sparsity and cost of human-written summaries in the news domain.
- Contributions: The proposed hierarchical model combines a pointer-generator network with an MMR module that adapts word-level attention using sentence-level relevance and redundancy scores.The MMR scores are integrated into the pointer-generator model to guide abstractive multi-document summarization.
- Contributions: The paper benchmarks multiple methods on Multi-News to establish baselines for large-scale MDS research.The authors also evaluate the model on existing MDS data, including DUC 2004.
2 Related Work
Prior MDS research spans extractive and abstractive traditional methods, while recent neural approaches have focused largely on single-document summarization. Related neural work adapts SDS models, hierarchical encoders, query relevance, or auxiliary supervision for MDS.
- Traditional methods: Traditional MDS methods include both extractive and abstractive approaches.The cited work covers extraction based on document content and abstraction-based generation.
- Neural summarization: Recent neural summarization research has advanced mainly in the single-document setting, using both extractive and abstractive methods.The related work situates neural MDS methods within broader progress in neural text summarization.
- Neural MDS: Several neural approaches address MDS by exploiting discourse graphs, auxiliary classification, or models adapted from single-document summarization.These methods represent different strategies for handling document clusters and limited MDS supervision.
- Neural MDS: Related SDS-to-MDS adaptations add document-level encoding or query relevance to sequence-to-sequence models.The cited examples include hierarchical encoding and relevance-aware encoder-decoder frameworks.
3 Multi-News Dataset
Multi-News is a large-scale news MDS dataset built from professionally written summaries and diverse cited sources. Its scale, longer summaries, and extractive-diversity characteristics support evaluation of abstractive multi-document summarization.
- Multi-News contains news articles and professionally written human summaries from newser.com, with links to cited originals.
- More than 1,500 source sites appear at least five times, and 20 editors contribute 85% of the summaries.
- The dataset includes examples with 2–10 source documents per summary and reflects settings where specialized events may have only a few articles.
- Multi-News has 44,972 training, 5,622 validation, and 5,622 test examples, making it two orders of magnitude larger than previous MDS news data.
- Summaries average about 260 words, testing fluent, concise, and coherent abstractive generation over generally longer outputs.
- 3.2 Diversity: Multi-News has abstractiveness comparable to SDS datasets and extractive characteristics similar to CNNDM, with variability in copied-word proportions and sequence lengths.
4 Preliminaries
The preliminaries introduce pointer-generator summarization, Transformer-based alternatives, and MMR as a relevance-and-novelty criterion for selecting sentences.
- Pointer-generator Network: Pointer-generator networks combine copying words from source documents with generating words from a fixed vocabulary.Attention produces a context vector, while a soft switch selects between vocabulary generation and source copying.
- Pointer-generator Network: At each decoding step, attention weights over encoder states represent relevant document content and contribute to the context vector.
- Transformer: Transformers replace recurrent layers with self-attention in encoder-decoder models and have been applied successfully to single-document summarization.
- MMR: MMR ranks candidate sentences by combining query relevance with information novelty and reducing redundancy.It ranks candidates using relevance to a query and redundancy relative to already selected sentences.
- Hi-MAP: Hi-MAP builds on a standard pointer-generator network by adding sentence-level representations and hidden-state-based MMR.
5 Hi-MAP Model
Hi-MAP extends pointer-generator summarization hierarchically to represent sentences and integrate learned MMR into attention, favoring relevant and nonredundant source content during decoding.
- Model Overview: Hi-MAP expands a pointer-generator network into a hierarchical network with an integrated MMR module for sentence-level scoring.
- Hierarchical Representations: The encoder derives each sentence representation from the final word-level hidden state and processes sentence representations with a sentence-level LSTM.
- Hierarchical Representations: The current decoded summary is represented as a single sentence using the decoder output at its last step.
- MMR-Attention: MMR ranks candidate sentence representations using similarity to the summary vector and learned self-attention over other candidates.The model normalizes candidate MMR scores with softmax.
- MMR-Attention: Learned transformations map candidate and summary representations into a common feature space for similarity calculation.
- MMR-Attention: Sentence-level MMR scores update word-level attention, assigning every token within a sentence the same sentence weight.
6 Experiments
The experiments compare extractive, pointer-generator, and Transformer baselines under established evaluation settings, while documenting input truncation and a limited non-neural baseline scope.
- Baselines and Methods: The study compares First-k, LexRank, TextRank, MMR, pointer-generator, and CopyTransformer systems.The neural comparisons include PG-Original, PG-MMR, PG-BRNN, and CopyTransformer.
- Experimental Setting: ROUGE evaluates unigram overlap, bigram overlap, and skip-bigram overlap with a maximum distance of four words.
- Experimental Setting: Neural inputs are truncated to 500 tokens by allocating tokens across source documents and concatenating the truncated documents.This setting effectively reduces multi-document summarization to single-document summarization on longer documents.
- Results: Table 5 reports ROUGE scores on DUC 2004 for models trained on CNN/Daily Mail data, following the setting of prior work.
- Results: Table 6 reports ROUGE scores for models trained and tested on Multi-News.
- Limitations: The experiments tested only several non-neural baselines because the study focused on deep methods for multi-document summarization.The authors leave implementation of additional classical methods for future work.
7 Analysis and Discussion
Experiments compare Multi-News systems across automatic and human evaluation. In-domain training improves performance, while Hi-MAP reduces redundancy and remains competitive with PG-MMR in informativeness and fluency.
- Extractive inputs: Using 500-token outputs from LexRank, TextRank, or MMR as abstractive inputs does not improve results.The authors believe the truncated input resembles the stronger First-3 baseline.
- Automatic evaluation: In-domain Multi-News training substantially improves model performance compared with cross-domain evaluation on DUC 2004.The authors attribute generally low cross-domain results to domain differences among DUC, CNNDM, and Multi-News.
- Automatic evaluation: The Transformer achieves the best R-1 score, whereas Hi-MAP outperforms it on R-2 and R-SU.
- Human evaluation: Human evaluation uses Best-Worst Scaling with pairwise comparisons of summaries shown the same truncated inputs as the tested systems.
- Human evaluation: Hi-MAP performs comparably to PG-MMR on informativeness and fluency but much better on non-redundancy in human evaluation.Human summaries score significantly better than all evaluated systems, indicating substantial room for improvement in readability and informativeness.
8 Conclusion
The paper introduces Multi-News, a large-scale multi-document news dataset, and an end-to-end MMR-enhanced pointer-generator model. The dataset and benchmarks support continued research, while the model performs competitively with prior multi-document summarization systems.
- Multi-News is introduced as the first large-scale multi-document news summarization dataset.
- The proposed end-to-end model incorporates MMR into a pointer-generator network and performs competitively with previous multi-document summarization models.
- The paper benchmarks methods on Multi-News and releases the dataset and code to promote future multi-document summarization research.