Source-linked AI summary
Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles
Yao Lu, Yue Dong, Laurent Charlin
TL;DR
Large-scale datasets for multi-document summarization are scarce, especially for abstractive modeling. The paper introduces Multi-XScience, which constructs a scientific-article task for generating related-work sections from abstracts and referenced papers, and reports that the dataset is challenging and well suited to abstractive models.
Problem
Large-scale datasets for multi-document summarization remain scarce, limiting the availability of domain-specific data for abstractive summarization.
Method
Multi-XScience constructs a scientific-article dataset whose task is to generate related-work paragraphs from a query abstract and reference-paper abstracts.
Results
Multi-XScience is challenging and better suited to abstractive summarization than previous multi-document summarization datasets.
Takeaways & Limitations
The dataset requires models to use high-level text understanding and abstraction capabilities rather than relying on positional or extractive biases.
Takeaways & Limitations
Reference documents are approximated with abstracts because full reference-paper text is often unavailable due to copyright restrictions.
Abstract
from arXiv · showhide
Multi-document summarization is a challenging task for which there exists little large-scale datasets. We propose Multi-XScience, a large-scale multi-document summarization dataset created from scientific articles. Multi-XScience introduces a challenging multi-document summarization task: writing the related-work section of a paper based on its abstract and the articles it references. Our work is inspired by extreme summarization, a dataset construction protocol that favours abstractive modeling approaches. Descriptive statistics and empirical results---using several state-of-the-art models trained on the Multi-XScience dataset---reveal that Multi-XScience is well suited for abstractive models.
1 Introduction
Multi-document summarization has fewer large-scale datasets than single-document summarization, particularly for abstractive modeling. Multi-XScience addresses this gap with a scientific-article task that generates related-work sections from an abstract and referenced papers.
- Multi-document summarization has progressed less than single-document summarization partly because large-scale datasets are scarce.
- Multi-XScience is a large-scale multi-document summarization dataset built from scientific articles.
- The task generates a paper’s related-work section from its abstract and the papers it references.
- The dataset follows extreme summarization and contains fewer positional and extractive biases than previous multi-document datasets.These properties make it less dependent on copying text from fixed source positions.
2 Multi-XScience Dataset
Multi-XScience is a large-scale scientific multi-document summarization dataset built from query papers and their references, with related-work paragraphs as targets. Its statistics and evaluations indicate high abstractiveness, reduced extractive and positional bias, and suitability for abstractive models.
- Dataset creation: The dataset combines arXiv papers with Microsoft Academic Graph references, using abstracts of query and cited papers as inputs.Construction included LaTeX cleaning, heuristic alignment, and five cleaning iterations interleaved with human verification.
- Dataset creation: The task generates a query paper’s related-work paragraphs from its abstract and the abstracts of cited articles.Only papers with explicit related-work sections are retained as query documents.
- Dataset creation: Paragraph targets keep inputs and outputs manageable while separating related-work passages into independent research topics.Whole related-work sections are avoided because their length challenges current models and their paragraphs often address different research threads.
- Dataset statistics and analysis: 60% more references than Multi-News and a high proportion of novel n-grams make Multi-XScience challenging and less extractive.Its related-work targets condense complicated ideas into short paragraphs, limiting the usefulness of copying source sentences.
- Dataset statistics and analysis: Lower positional and extractive bias means models cannot achieve high scores simply by copying sentences or exploiting fixed source positions.These properties indicate that the dataset requires models to understand source text and supports abstractive summarization.
- Human evaluation: 2.82±0.4 (95% C.I.) was the average human-evaluated quality score, indicating substantial overlap between reference abstracts and related-work targets.Two PhD students evaluated 25 randomly selected test pairs using the study’s quality scale.
3 Experiments & Results
The experiments evaluate extractive and abstractive models on Multi-XScience using automatic, human, and qualitative analyses. Results indicate that the dataset favors abstractive generation, although current models remain challenged.
- Models and evaluation: The study evaluates lead, extractive-oracle, LexRank, TextRank, HiMAP, HierSumm, Pointer-Generator, BART, BertABS, and SciBert-based models.Single-document models are adapted by concatenating reference inputs; Pointer-Generator uses attention-based copying, while BART and BertABS use pretrained encoders or encoder-decoder components.
- Automatic evaluation: Most abstractive models outperform unsupervised extractive baselines by wide margins and significantly outperform the extractive oracle in ROUGE-L.The comparison is reported as evidence that Multi-XScience is suitable for abstractive summarization.
- Automatic evaluation: Pointer-Generator outperforms self-pretrained abstractive models such as BART and BertABS, while SciBertAbs achieves higher ROUGE-L than BART.Pointer-Generator produces highly abstractive summaries, whereas BART has the lowest novel n-gram proportion; the authors associate this with domain shift and supervised-data requirements.
- Human evaluation: Human rank scores are 1.54 for ext-oracle, 2.28 for HiMAP, and 2.18 for Pointer-Generator.Two judges ranked 25 randomized samples on a 1-to-3 scale, with higher scores indicating better generation quality.
- Qualitative analysis: Human feedback rates abstractive models’ writing style as much better than extractive models, while examples show generation models match related-work style and correct content.The extractive oracle copies source text and captures content but fails to resemble related-work writing.
4 Related Work
Scientific document summarization has lacked large-scale datasets suitable for neural models. The paper positions Multi-XScience as a response to this gap while also providing generation examples for related-work summarization.
- Scientific document summarization: Scientific summarization is challenging, but existing datasets have generally been small and insufficient in scale for neural-based model training.Examples include CL-Scisumm with 30–50 articles and human-annotated datasets with at most 1,000 article-summary pairs.
- Scientific document summarization: The paper states that the lack of large-scale datasets slowed development of multi-document summarization methods.It presents the proposed dataset as intended to address this development gap.
- Generation examples: Table 8 compares generation examples from the extractive oracle, HiMAP, and Pointer-Generator.The comparison illustrates how systems produce related-work summaries.
5 Extensions of Multi-XScience
Multi-XScience is designed primarily for multi-document summarization but may support additional scientific-document understanding tasks. The paper identifies graph information and unsupervised in-domain corpora as extensions to explore.
- Graph-based summarization: The dataset could support graph-based summarization by combining text with citation-graph information from MAG.The proposed extension uses graph information such as citation links alongside plain text inputs.
- In-domain corpus: An unsupervised in-domain corpus of related scientific work could be explored for downstream scientific-document understanding tasks.The paper suggests using papers from an N-hop MAG subgraph as a possible corpus.
6 Conclusion
The paper introduces Multi-XScience, a large-scale scientific-article dataset for multi-document summarization. Its task and experiments indicate suitability for abstractive models, while current models still find the task challenging.
- Contribution: Multi-XScience addresses the lack of large-scale datasets that has slowed multi-document summarization research.The dataset is constructed from scientific articles.
- Conclusion: The task requires models to summarize a paper’s related work from its abstract and referenced articles, demanding text understanding and abstraction.Experimental results show the dataset is amenable to abstractive models and challenging for current models.