Source-linked AI summary
Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization
Logan Lebanoff, Kaiqiang Song, Fei Liu
TL;DR
Multi-document summarization lacks the abundant parallel data available for single-document summarization, creating a need to adapt pretrained encoder-decoder models. The paper combines maximal marginal relevance with a pointer-generator network to select representative sentences and fuse them into abstractive summaries. PG-MMR compares favorably with extractive and abstractive baselines under automatic metrics and human judgments, without requiring multi-document training data.
Problem
Multi-document summarization has scarce, costly parallel data and has not fully benefited from neural encoder-decoder models developed for single-document summarization.
Method
PG-MMR combines maximal marginal relevance for selecting important multi-document sentences with a pointer-generator network for abstractive sentence fusion.
Results
PG-MMR compares favorably with state-of-the-art extractive and abstractive systems on automatic metrics and human judgments.
Takeaways & Limitations
The adaptation extends a single-document abstractive model to multi-document inputs without requiring training on multi-document data.
Abstract
from arXiv · showhide
Generating a text abstract from a set of documents remains a challenging task. The neural encoder-decoder framework has recently been exploited to summarize single documents, but its success can in part be attributed to the availability of large parallel data automatically acquired from the Web. In contrast, parallel data for multi-document summarization are scarce and costly to obtain. There is a pressing need to adapt an encoder-decoder model trained on single-document summarization data to work with multiple-document input. In this paper, we present an initial investigation into a novel adaptation method. It exploits the maximal marginal relevance method to select representative sentences from multi-document input, and leverages an abstractive encoder-decoder model to fuse disparate sentences to an abstractive summary. The adaptation method is robust and itself requires no training data. Our system compares favorably to state-of-the-art extractive and abstractive approaches judged by automatic metrics and human assessors.
1 Introduction
Multi-document summarization lacks the abundant parallel data and abstraction capabilities that support neural single-document summarization. The paper therefore adapts a single-document encoder-decoder model using representative-sentence selection and evaluates it against established systems.
- Motivation: Multi-document summarization has not fully benefited from neural encoder-decoder models, despite practical applications across document sets.MDS condenses documents likely written by multiple authors into short, informative summaries.
- Motivation: Existing multi-document systems are mostly extractive and therefore cannot paraphrase, generalize, or synthesize content abstractively.This limits their ability to produce genuinely abstractive summaries.
- Challenges: Applying a single-document model to a concatenated mega-document can miss important content because summary-worthy information is not concentrated at the beginning.The mega-document also contains redundant text that can be reused during generation.
- Approach: PG-MMR selects important sentences from multi-document input with maximal marginal relevance and uses a pointer-generator model to generate abstractive summaries.The method modifies attention toward selected sentences and reduces selection likelihood for sentences similar to the partial summary.
- Contributions: The study investigates adapting the encoder-decoder framework from single- to multi-document summarization and compares the method with extractive and abstractive systems using automatic and human evaluation.The authors describe this as an initial investigation and report favorable comparisons.
2 Related Work
Prior multi-document summarization research has emphasized extractive methods, while abstractive neural work has mainly targeted single documents because suitable multi-document training data are limited. This paper addresses that gap by combining pointer-generator networks with maximal marginal relevance.
- Extractive and neural approaches: Popular multi-document summarization methods extract important sentences and optionally compress them into summaries.Neural representations have also been explored for both single- and multi-document summarization.
- Earlier abstractive work: Earlier abstractive systems used domain templates, text planning or open information extraction, and natural language generation on small domain-specific datasets.Their experiments were limited by the availability of labelled data.
- Neural abstractive work: Neural abstractive encoder-decoder studies have focused primarily on single-document summarization because training data are abundant.This emphasis leaves multi-document abstraction comparatively underexplored.
- Related extensions: Only a few studies had extended abstractive systems toward query-focused or hierarchical multi-document summarization.The paper situates its approach among these limited efforts.
- This work: The paper combines pointer-generator networks trained on single-document data with maximal marginal relevance to generate abstractive summaries without multi-document training.The adaptation targets important-content recognition in multi-document inputs.
3 Limits of the Encoder-Decoder Model
The encoder-decoder model can generate and copy words, but its single-document training and position-based attention do not reliably identify salient, non-redundant content in multi-document inputs. These limitations motivate an external sentence-selection mechanism.
- Encoder-decoder architecture: The encoder represents input words with hidden states, while a unidirectional decoder generates the summary one word at a time.The input is treated as a word sequence for sentence and single-document summarization.
- Attention mechanism: Attention weights measure the importance of each input word for generating each output word, using decoder and encoder states plus cumulative attention.Cumulative attention lowers the likelihood of reusing a word that has already contributed to earlier output.
- Generation: The context vector summarizes the input, and the decoder uses it with its hidden state to compute vocabulary-word probabilities.The context vector is a weighted sum of encoder hidden states.
- Pointer-generator: The pointer-generator switch combines vocabulary generation with copying from the input, where a word’s copy probability sums attention over its occurrences.The model’s final word probability is a weighted combination of vocabulary and copy probabilities.
- Multi-document limitations: Training on single-document data does not equip the model to identify salient multi-document content, while position-based attention can repeatedly use semantically redundant text.The paper proposes an external model to select representative sentences and dynamically adjust attention.
4 Our Method
PG-MMR adapts a single-document pointer-generator to multi-document summarization by using MMR to select salient, nonredundant source sentences and modifying attention at test time. It estimates sentence importance from single-document data, then iteratively guides abstractive generation without retraining the neural model.
- Maximal marginal relevance: MMR selects source sentences that are important relative to the document while minimizing overlap with the partial summary.Sim1 measures document similarity as an importance proxy, while maximum similarity to summary sentences measures redundancy; λ balances the two.
- Iterative generation: PG-MMR selects the K highest-scored source sentences, generates one summary sentence, and updates importance and redundancy scores iteratively.Sentences similar to the partial summary receive lower scores, helping subsequent generations focus on salient content not already covered.
- Attention control: At test time, muting forces words outside the selected sentences to receive zero attention, then renormalizes the remaining attention weights.This allows the pretrained PG system to use selected multi-document content without retraining.
- Attention control: An alternative attention variant multiplies word salience by sentence salience and renormalizes the resulting weights before predicting the next summary word.The original muting variant changes attention by suppressing unselected sentences, whereas this variant also reweights selected content by sentence salience.
- Sentence importance: Sentence importance is estimated with a supervised regression model trained on abundant single-document data and applied to multi-document inputs.The model uses sentence length, absolute and relative position, sentence quality, and document-sentence similarity; sentence quality is represented with PG encoder vectors and document similarity uses cosine similarity.
- Algorithm: The PG-MMR algorithm trains PG on single-document data, scores multi-document sentences, selects high-scoring sentences, and updates redundancy after each generated sentence.Redundancy uses ROUGE-L precision between a source sentence and the partial summary, lowering the MMR score of overlapping sentences.
5 Experimental Setup
The experiments evaluate PG-MMR on standard multi-document datasets where ten chronologically ordered documents are concatenated into a mega-document and summarized in at most 100 words. Comparisons include extractive and abstractive baselines, with ROUGE used for automatic evaluation.
- Datasets: PG-MMR is tested on DUC-03, DUC-04, TAC-08, TAC-10, and TAC-11, whose topics contain 30, 50, 48, 46, and 44 document sets respectively.Each topic contains ten documents discussing a topic, and the system generates a concise summary of 100 words or less.
- Datasets: The ten documents in each set are chronologically ordered and concatenated into a mega-document for PG-MMR input.Sentences beginning with quotation marks or not ending with periods are excluded.
- Training: The PG model is trained on 287,226 CNN/Daily Mail single-news articles paired with human-written highlights.Articles average 781 tokens and summaries average 56 tokens, or 3.75 sentences.
- Baselines: Baselines cover unsupervised extractive systems, supervised extractive systems, and abstractive systems including Opinosis, Extract+Rewrite, and the original pointer-generator.The extractive baselines include SumBasic, KL-Sum, LexRank, Centroid, ICSISumm, and DPP.
6 Results
PG-MMR was evaluated against extractive and abstractive baselines using automatic metrics, human judgments, content-location analysis, and extractiveness analysis. It generally outperformed competing systems while searching more broadly across multi-document inputs.
- Evaluation: PG-MMR was evaluated against baselines on standard multi-document summarization datasets using automatic metrics and human assessors.The experiments included ROUGE evaluation and Mechanical Turk judgments of linguistic quality.
- Automatic evaluation: PG-MMR surpassed all unsupervised extractive baselines and outperformed existing abstractive systems on ROUGE.Its R-2 F-scores were 5.03/6.03/8.73 for DUC-04 and 5.12/6.40/10.92 for TAC-11 against the reported systems.
- Content location: PG-MMR searched a broader range of source sentences than PG-Original, whose summary n-grams concentrated near the beginning of the input.For PG-MMR, n-grams from the first and second summary sentences came from source-sentence ranges of (2, 44) and (6, 53), respectively.
- Degree of extractiveness: Both PG-MMR and PG-Original were highly extractive, with about 90% of summary n-grams appearing in the input.About 52% and 41% of their summary sentences did not appear in the source, while Extract+Rewrite showed greater abstraction.
- Human evaluation: PG-MMR received the highest ratings for informativeness and non-redundancy, while LexRank received the highest fluency rating.PG-MMR summaries were also frequently ranked first or second overall, outperforming the other systems.
- Example summaries: PG-MMR generated summaries that effectively condensed important source content, whereas PG-Original and Extract+Rewrite showed content-selection or redundancy limitations.Extract+Rewrite tended to produce short, title-like sentences because it was trained with seven-word target summaries.
7 Conclusion
The PG-MMR method combines extractive sentence selection with abstractive fusion to summarize multi-document inputs. It achieves competitive results against strong extractive and abstractive baselines.
- PG-MMR combines MMR sentence extraction with a neural pointer-generator model to fuse source sentences into abstractive summaries.MMR identifies important source sentences, while PG generates the abstract.
- The method generates abstractive summaries from multi-document inputs.
- PG-MMR outperforms strong extractive and abstractive baselines.