Source-linked AI summary
ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R. Fabbri, Irene Li, Dan Friedman, Dragomir R. Radev
TL;DR
Scientific paper summarization lacks large annotated corpora and should capture both authors’ messages and papers’ community impact. The paper releases a 1,000-example corpus and hybrid models combining abstracts with citation-informed text. Experiments show that the corpus improves neural summarization and that hybrid summaries are more comprehensive than abstracts and citation-only alternatives.
Problem
Scientific paper summarization is difficult because papers are long and technical, existing datasets are small, and abstracts may omit impacts recognized by the research community.
Method
The paper constructs a manually annotated corpus of 1,000 papers and develops hybrid models that combine each paper’s abstract with cited text spans and citation authority information.
Results
The larger corpus enabled neural models to outperform prior shared-task systems, while hybrid summaries were more comprehensive than abstracts and citation-based summaries.
Takeaways & Limitations
The corpus and hybrid methods provide a framework for supervised scientific summarization that represents both author and research-community perspectives.
Abstract
from arXiv · showhide
Scientific article summarization is challenging: large, annotated corpora are not available, and the summary should ideally include the article's impacts on research community. This paper provides novel solutions to these two challenges. We 1) develop and release the first large-scale manually-annotated corpus for scientific papers (on computational linguistics) by enabling faster annotation, and 2) propose summarization methods that integrate the authors' original highlights (abstract) and the article's actual impacts on the community (citations), to create comprehensive, hybrid summaries. We conduct experiments to demonstrate the efficacy of our corpus in training data-driven models for scientific paper summarization and the advantage of our hybrid summaries over abstracts and traditional citation-based summaries. Our large annotated corpus and hybrid methods provide a new framework for scientific paper summarization research.
Introduction
Scientific paper summarization is hindered by papers’ length, technical complexity, citations, and small annotated datasets. ScisummNet addresses these gaps with a 1,000-paper corpus and hybrid summaries combining author and community perspectives.
- Scientific papers are longer, more technical, section-structured, and citation-rich than news articles, making summarization less explored and more difficult.
- Existing scientific summarization datasets are small, limiting data-driven approaches such as neural networks.
- A citation analysis can reveal important community-valued contributions absent from an abstract, such as Bergsma and Lin’s noun gender dataset.
- The proposed hybrid models combine abstracts with cited text spans to represent both authors’ highlighted content and the research community’s perceived impact.
- On CL-SciSumm, the larger corpus enabled neural models to outperform prior shared-task participants, while hybrid summaries were more comprehensive than abstracts.
Background & Motivation
Prior work uses extractive and citation-based methods, but citation-focused summaries can omit authors’ original messages. The paper therefore combines author-provided abstracts with community-informed cited spans.
- Background: Neural extractive models rank and select sentences, extending approaches that have succeeded in news summarization to scientific papers.
- Citation-based summarization: Citation sentences summarize a reference paper from citing authors’ perspectives and collectively reflect its research-community impact.
- Citation-based summarization: Cited text span methods identify passages in the reference paper that incoming citations discuss, retaining the paper’s wording while reflecting community insights.
- Citation-based summarization: Cited text span-based summarization outperformed direct summarization of citing sentences and became the default approach in TAC 2014 and CL-SciSumm.
- Motivation: Citation-based summaries may overlook authors’ original messages, especially motivation, while cited-span summaries can fall short of abstracts in ROUGE evaluation.
- Motivation: Scientific summarization datasets generally contain only several dozen articles, restricting supervised and neural approaches.
Dataset Construction
The corpus scales expert annotation by combining abstracts with selected incoming citation sentences. Validation indicates that this procedure captures most major points while substantially reducing annotation time.
- Corpus: The corpus contains 1,000 most-cited ACL Anthology papers, citation information, and expert-annotated gold summaries.
- Citation selection: For each reference paper, the dataset samples and cleans 20 citation sentences while preserving the oldest and latest citations.
- Annotation: Five NLP-expert annotators read each paper’s abstract and incoming citations, then select salient citations and write summaries from those materials.
- Annotation: The annotation procedure avoids reading the full paper while using citations to add major contributions absent from the abstract.
- Validation: Over 90% of major points were covered using abstracts plus citing sentences, requiring 30% of the time needed for full-paper reading.
- Validation: The resulting summaries averaged 151 words, and salient-citation annotation achieved a Cohen’s kappa of 0.75.
Hybrid Summarization Models
The hybrid models combine an abstract with citation-derived text spans so summaries reflect both authors’ perspective and the research community’s perceived impact. They rank these inputs using graph-based sentence representations and salience scores, then select non-redundant sentences with greedy heuristics.
- Hybrid Summarization Models: Hybrid summarization combines the reference paper’s abstract with cited text spans identified from incoming citation sentences.The abstract represents the authors’ perspective, while cited spans represent community insights.
- Hybrid Summarization Models: Hybrid 1 summarizes the union of the abstract and cited text spans, whereas Hybrid 2 augments the abstract with salient text extracted from cited spans.Hybrid 2 is motivated by preserving the authors’ concise self-summary while adding community views it omits.
- Hybrid Summarization Models: For each incoming citation sentence, the system selects the two most tf-idf-cosine-similar sentences in the reference paper and unions them across citations.The resulting abstract-plus-citation input contains about 40 sentences on average in the experiments.
- Hybrid Summarization Models: The sentence relation graph uses tf-idf cosine similarity as edge weights, while each cited span receives an authority feature based on citing-paper citation counts.Abstract sentences receive the reference paper’s citation count.
- Hybrid Summarization Models: An LSTM encodes each sentence, and a GCN propagates information over the sentence relation graph to produce globally contextualized sentence embeddings.The GCN takes sentence embeddings as node features and incorporates graph structure through multiple propagation layers.
- Hybrid Summarization Models: The model estimates normalized salience scores for input sentences and trains them against ROUGE-derived target distributions using cross-entropy loss.Targets average each sentence’s ROUGE-1 and ROUGE-2 scores against the gold summary before rescaling.
Experiments
The experiments evaluate the corpus for data-driven summarization and compare hybrid summaries with abstract and citation-based baselines. Results show that larger training data improves neural performance, while hybrid summaries better combine author and community perspectives.
- Experimental setup: The experiments assess the proposed corpus and hybrid methods using CL-SciSumm benchmarks, ROUGE metrics, and qualitative and human evaluation.Models are trained and tested on established CL-SciSumm splits, with experiments comparing corpus training effects and hybrid summaries against baselines.
- Experiment 1: Corpus usefulness: 1,000 training examples enable GCN Cited text spans to outperform all models trained on the existing 30-example CL-SciSumm corpus.The model gains significant boosts across ROUGE metrics, including +5 in ROUGE-3-F, relative to models trained on CL-SciSumm.
- Experiment 2: Hybrid methods: Both hybrid models outperform pure cited text span summaries, and Hybrid 1 surpasses abstracts in ROUGE-2 Recall and ROUGE-2 F1.Hybrid 2 performs better than Hybrid 1 and adds two sentences on average to the original abstract.
- Qualitative analysis: Hybrid summaries combine technical contributions influential to the community with author motivations and objectives that citation-based summaries may omit.In the qualitative example, the hybrid summary augments the abstract with salient cited spans, while the cited-span summary provides technical details but lacks some original messages.
- Human evaluation: Human evaluators scored hybrid summaries 4.5 for coverage and 4.2 for coherence on a 1–5 scale.The authors report room for improving coherence while describing the summaries as comprehensive and readable.
- Authority feature: Adding the authority feature yields moderate improvements across the models.The feature is intended to reflect the authority of each citation made by the research community.
Conclusion
The paper introduces a 1,000-example manually annotated corpus and hybrid summarization methods that integrate authors’ and research community’s insights. Experiments find that the corpus supports effective neural training and the hybrid models produce more comprehensive summaries than abstract-only and citation-based approaches.
- Conclusion: The corpus contains 1,000 examples of papers, citation information, and human summaries, making it orders of magnitude larger than prior datasets.It is designed to facilitate future supervised research in scientific paper summarization.
- Conclusion: The hybrid methods integrate authors’ insights with the research community’s insights to address limitations of abstracts and traditional citation-based summaries.Abstracts may miss actual impacts, while citation-based summaries may overlook authors’ original messages.
- Conclusion: Experiments show that the proposed dataset effectively trains data-driven neural models and that hybrid models produce more comprehensive summaries than both comparison approaches.The comparison approaches are abstracts and traditional citation-based summaries.