Source-linked AI summary
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, Nazli Goharian
TL;DR
Existing neural abstractive summarization methods mainly address relatively short documents, leaving long, structured documents such as scientific papers less served. The paper introduces a discourse-aware hierarchical encoder-decoder for this setting and reports significant improvements over state-of-the-art abstractive methods on arXiv and PubMed datasets. The authors note that ROUGE does not capture all qualities of long-document summaries.
Problem
Existing datasets and neural abstractive summarization models mainly focus on relatively short documents, whereas scientific papers are longer and structurally organized.
Method
The model combines a hierarchical encoder of discourse structure with a discourse-aware decoder and a pointer-generator mechanism for abstractive scientific-paper summarization.
Results
The model significantly outperforms state-of-the-art abstractive methods on both large-scale scientific-paper datasets, with ROUGE-1 about 4 and 3 points higher than Pntr-Gen-Seq2Seq on arXiv and PubMed, respectively.
Takeaways & Limitations
The results support neural abstractive summarization of long, structured scientific papers using discourse-aware modeling.
Takeaways & Limitations
ROUGE evaluation does not capture nuances of summary coherence or coverage, motivating expert human evaluations.
Abstract
from arXiv · showhide
Neural abstractive summarization models have led to promising results in summarizing relatively short documents. We propose the first model for abstractive summarization of single, longer-form documents (e.g., research papers). Our approach consists of a new hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary. Empirical results on two large-scale datasets of scientific papers show that our model significantly outperforms state-of-the-art models.
1 Introduction
Existing summarization datasets and neural models mainly target relatively short documents, while scientific papers are longer and follow structured discourse. The paper proposes an abstractive model designed for this long-document setting.
- Existing large-scale datasets and neural summarization models primarily focus on relatively short documents such as news articles.CNN/Daily Mail articles average about 600 words, while longer documents remain less addressed.
- Scientific papers are long, structured documents whose discourse commonly progresses through problems, methods, experiments/results, and conclusions.
- Abstractive summarization can generate novel words and phrases, offering an alternative to predominantly extractive approaches.
- Seq2seq models struggle with long sequences because each decoding step must construct a context vector from all source tokens.
- The proposed model uses a discourse-structured hierarchical encoder and discourse-aware decoder for abstractive scientific-paper summarization.The authors also introduce large-scale arXiv and PubMed datasets, and report outperforming state-of-the-art summarization models.
2 Background
The background presents seq2seq abstractive summarization as encoder-decoder generation with attention-based context vectors. Attention scores weight encoder states to provide source information at each decoding step.
- In seq2seq summarization, an encoder RNN represents the input document and a decoder RNN generates the summary one token at a time.
- Figure 1 shows word-level and section-level RNNs feeding attention, a decoder RNN, and a predict network for joint pointer-generator output.
- The decoder predicts each output using its input, hidden state, and information about the encoded source sequence.
- Attentive decoding: Attention maps decoder and encoder states to a context vector computed as a weighted sum of encoder states.
- Attentive decoding: Attention weights are calculated over encoder states for each decoding timestep.
- Attentive decoding: The model uses an additive attention score with a weight vector and linear mappings.
- Attentive decoding: The additive scoring function includes weight matrices W1 and W2 together with a bias vector b.
3 Model
The model uses a hierarchical encoder to represent discourse sections and a discourse-aware decoder to attend to relevant sections while generating summaries. It also supports source copying and coverage tracking for long documents.
- Encoder: The hierarchical encoder first represents each discourse section, then encodes the document from those section representations.The section encoder parameters are shared across sections, and both section- and document-level encoders use single-layer bidirectional LSTMs.
- Discourse-aware decoder: The decoder attends to relevant discourse sections and uses that information to modify word-level attention.This produces a context vector that incorporates both word-level and section-level attention.
- Copying from source: The decoder can either generate a word from its vocabulary or copy a word directly from the source.A binary variable indicates whether each output word is generated or copied; copied-word probability is based on source attention weights.
- Decoder coverage: Coverage tracking sums attention from previous timesteps and supplies it to attention to reduce repeated attention and phrase generation.The coverage vector also contains information about which document discourse sections have already been attended to.
4 Related work
Prior summarization work largely emphasizes extractive methods or abstractive models for shorter documents, while scientific-document approaches often rely on extraction and external citations. This paper instead addresses long scientific papers with an abstractive model and datasets designed for that setting.
- Summarization methods: Prior summarization research includes frequency-based, graph-based, topic-modeling, and neural extractive approaches.Abstractive summarization is presented as an alternative that can produce novel words and phrases.
- Long-document focus: Existing neural abstractive models commonly target relatively short news articles, whereas this work addresses longer structured documents.The related-work discussion distinguishes the paper's long-document focus from prior short-document settings.
- Model distinction: The paper differs from a related sentence-level attention model by using a hierarchical encoder for document structure.The supplied passage introduces this distinction but does not provide the full comparison.
- Scientific-document summarization: Scientific-document summarization approaches described here are extractive and rely on external information such as citations.The paper notes that such external information may not be available for all papers.
5 Data
The datasets use scientific papers because they are long, follow standard discourse structures, and include abstracts that can serve as ground-truth summaries. The resulting arXiv and PubMed corpora are substantially larger in document and summary length than common news datasets.
- Dataset motivation: Scientific papers provide long documents, standard discourse structure, and abstracts that support supervised summarization training.The dataset construction follows the use of news abstracts as ground-truth summaries.
- Filtering and structure: The data pipeline removes excessively long or short documents and papers lacking abstracts or discourse structure.It uses level-1 section headings as discourse information.
- Preprocessing: The pipeline converts arXiv LaTeX to plain text, preserves discourse sections, removes figures and tables, and normalizes formulas and citations.Sections after the conclusion are also removed according to the supplied passage.
- Dataset statistics: The arXiv and PubMed datasets have substantially larger document and summary lengths than existing large-scale summarization datasets.About 3% of PubMed and 5% of arXiv are used for validation, with similar proportions for testing.
6 Experiments
The experiments evaluate the model on arXiv and PubMed scientific-paper datasets using ROUGE and compare it with extractive and abstractive baselines. The model significantly outperforms state-of-the-art abstractive methods and captures more diverse discourse information in generated summaries.
- Setup and comparison: ROUGE evaluates the model against extractive baselines and state-of-the-art abstractive systems on the arXiv and PubMed datasets.The comparison includes LexRank, SumBasic, LSA, Attn-Seq2Seq, and Pntr-Gen-Seq2Seq.
- Results: About 4 and 3 ROUGE-1 points separate the model from Pntr-Gen-Seq2Seq on arXiv and PubMed, respectively.The paper reports these improvements as significant.
- Results: The model significantly outperforms state-of-the-art abstractive methods on both datasets and surpasses most extractive methods.LexRank remains higher on one ROUGE score.
- Results: Extractive methods can achieve higher ROUGE because they copy salient sentences from the document.This provides context for interpreting comparisons between extractive and abstractive systems.
- Discourse analysis: Compared with Pntr-Gen-Seq2Seq, the model's summaries include more methodology and impact information rather than focusing mostly on the problem.The paper attributes this to a context vector that attends to various discourse sections.
7 Conclusions and future work
The paper presents a first attempt at neural abstractive summarization for single, long documents and reports encouraging results on structured scientific papers. It also identifies limitations in evaluation and calls for further work on the task.
- Conclusions: The work presents the first neural abstractive summarization approach for single, long documents such as scientific papers.Its new datasets are intended to support further exploration of this challenging problem.
- Limitations and future work: ROUGE does not capture nuances of summary coherence or coverage, motivating future expert human evaluations.The paper describes evaluating these qualities for long-document summarization as non-trivial.