Source-linked AI summary
Leveraging Graph to Improve Abstractive Multi-Document Summarization
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, Junping Du
TL;DR
MDS needs better ways to represent cross-document relations and organize long, coherent summaries. The paper integrates explicit similarity, topic, and discourse graphs into neural encoding and decoding, combines the model with pretrained language models, and reports substantial improvements over strong baselines on WikiSum and MultiNews. Its results support graph modeling as useful for abstractive MDS, while the reported evaluation notes limit the benefit of simply using longer inputs.
Problem
MDS must represent multiple documents containing repeated, complementary, or contradictory information while detecting salient content and organizing long summaries.
Method
The model uses explicit graph representations in graph-informed document encoding and hierarchical graph-guided summary generation, and combines this architecture with pretrained language models.
Results
The model brings substantial improvements over several strong baselines on both WikiSum and MultiNews, while graph modeling significantly improves pretrained language models on MDS.
Takeaways & Limitations
Explicit graphs help the model capture cross-document relations and organize summaries, while pretrained language models extend the approach to much longer inputs.
Takeaways & Limitations
The evaluation notes that longer inputs do not achieve obvious improvements, and results are reported only for GraphSum+RoBERTa because XLNet and BERT perform worse than RoBERTa.
Abstract
from arXiv · showhide
Graphs that capture relations between textual units have great benefits for detecting salient information from multiple documents and generating overall coherent summaries. In this paper, we develop a neural abstractive multi-document summarization (MDS) model which can leverage well-known graph representations of documents such as similarity graph and discourse graph, to more effectively process multiple input documents and produce abstractive summaries. Our model utilizes graphs to encode documents in order to capture cross-document relations, which is crucial to summarizing long documents. Our model can also take advantage of graphs to guide the summary generation process, which is beneficial for generating coherent and concise summaries. Furthermore, pre-trained language models can be easily combined with our model, which further improve the summarization performance significantly. Empirical results on the WikiSum and MultiNews dataset show that the proposed architecture brings substantial improvements over several strong baselines.
1 Introduction
The paper addresses the difficulty of representing cross-document relations and organizing long summaries in MDS by integrating explicit graphs into neural encoding and decoding. Experiments show substantial improvements across datasets and strong baselines, while graph modeling also complements pretrained language models.
- Motivation: MDS requires cross-document links to detect salient information, redundancy, and coherence across repeated, complementary, or contradictory content.These challenges distinguish MDS from single-document summarization.
- Approach: The model incorporates explicit similarity, topic, and discourse graphs into document representation and summary generation.Graph-informed attention captures cross-document relations, while hierarchical graph attention guides summary content organization.
- Pretrained language models: The graph model combines with pretrained language models, including BERT, RoBERTa, and XLNet, to process much longer inputs and improve MDS performance.The combined model adopts advantages from both graph modeling and pretrained language models.
- Approach: Explicit graph representations benefit both document representation and summary generation.The architecture is designed to use graphs throughout the encoding and decoding processes.
- Results: The model brings substantial improvements over several strong baselines on both WikiSum and MultiNews.Analysis also reports better performance for longer inputs and greater benefits from graphs with richer relations.
2 Related Work
Earlier MDS work largely relied on graph-based extraction or limited abstractive strategies, while neural Seq2Seq approaches did not study cross-document graph relations directly. This paper instead integrates explicit graphs into neural encoding and decoding and combines them with pretrained language models for longer inputs.
- Extractive MDS: Most previous MDS approaches are extractive, ranking salient textual units with graph-based representations of sentences.LexRank and related methods use lexical similarity or other document-level graph relations for selection.
- Abstractive MDS: Earlier abstractive MDS approaches used sentence fusion, information extraction, paraphrasing, or AMR-based generation, but had limited success.These approaches represent several distinct strategies for abstractive summary construction.
- Neural abstractive MDS: Neural abstractive models developed for single-document summarization were not straightforward to extend to MDS because of limited training data.Earlier approaches transferred single-document models or used unsupervised reconstruction objectives.
- Research gap: Previous Seq2Seq approaches did not study the importance of cross-document relations and graph representations in MDS.Some combined extractive models with standard Seq2Seq generation without directly addressing graph relations.
- Paper's distinction: Unlike prior work that simply replaced attention weights with a graph matrix, this model uses graph-informed attention for encoding and graph structure for summary decoding.The paper also combines its graph model with pretrained language models to process much longer inputs effectively.
3 Model Description
The model extends an encoder-decoder architecture with graph-aware encoding and decoding for paragraph-level multi-document inputs. Explicit graph relations enrich document representations, guide summary generation, and support processing longer inputs with pre-trained language models.
- Input and graph construction: Source documents are split into paragraphs, and a graph matrix G records relation weights between paragraph pairs.Similarity graphs can be constructed from cosine similarities between paragraph tf-idf representations.
- Graph encoding: The encoder stacks token-level Transformer layers with paragraph-level graph encoding layers that model documents globally.Graph encoding layers extend Transformer self-attention with explicit graph information.
- Graph encoding: Graph-informed self-attention combines latent attention dependencies with Gaussian biases derived from explicit graph relations.The bias acts as a prior constraint, while σ controls the influence intensity of the graph structure.
- Graph decoding: The decoder uses hierarchical graph attention, combining global paragraph-level graph attention with local token-level normalized attention.Global attention regularizes paragraph attention using graph structure, while local attention captures within-paragraph context.
- Graph decoding: A central paragraph position is predicted for each summary token, after which graph relations regularize attention over the remaining paragraphs.This aligns source information and summary output globally during decoding.
- Pre-trained language models: The graph model can combine with pre-trained language models by formatting each paragraph as a separate [CLS]–[SEP] segment.Graph modeling supports much longer inputs than the 512-token limit used by BERT and RoBERTa.
4 Experiments
Experiments on WikiSum and MultiNews evaluate graph-based abstractive MDS across graph types, input lengths, ablations, pretrained language models, and human judgments. GraphSum consistently improves over strong baselines, with benefits increasing for longer inputs and richer graph relations.
- Datasets: WikiSum and MultiNews provide evaluation settings with long, multi-document inputs and different corpus characteristics.WikiSum uses ranked paragraphs, while MultiNews contains news articles from over 1,500 sources.
- Automatic Evaluation: GraphSum achieves significant improvements over strong abstractive baselines on all three WikiSum metrics.The comparison includes Flat Transformer, T-DMCA, and Hierarchical Transformer baselines.
- Automatic Evaluation: GraphSum consistently outperforms all baselines on MultiNews, and GraphSum+RoBERTa significantly improves all reported metrics over RoBERTa+FT.This extends the reported gains to a different type of multi-document corpus and pretrained-language-model setting.
- Graph Types: Discourse graphs achieve the best performance on ROUGE-2 and ROUGE-L, while topic graphs outperform similarity graphs on ROUGE-1 and ROUGE-2.The comparison supports greater benefits from graphs representing richer relations.
- Input Length: GraphSum’s advantage over Hierarchical Transformer tends to grow as WikiSum input length increases from 500 to 3000 tokens.The results associate graph modeling with processing longer inputs while maintaining better performance.
- Ablation Study: Ablations show that graph-informed encoding and graph-guided decoding are both beneficial for multi-document summarization.Removing either graph encoder or graph decoder component reduces the demonstrated contribution of graph modeling.
- Human Evaluation: Human evaluation finds GraphSum and GraphSum+RoBERTa produce more salient, fluent, and concise summaries than other evaluated models.Three annotators ranked outputs from 50 WikiSum and 50 MultiNews test instances.
5 Conclusion
The paper proposes graph-based neural abstractive MDS to capture richer relations in long inputs and guide summary decoding. Combined with pretrained language models, the approach improves summarization performance over strong baselines, while future work considers richer graph representations.
- Contributions: The model incorporates explicit graph representations into document encoding to capture richer relations within long inputs.It also uses graph structure to guide summary decoding.
- Contributions: Combining the graph model with pretrained language models further improves multi-document summarization performance.The paper presents this combination as an effective way to process long inputs.
- Conclusion: Experiments show that the proposed model outperforms several strong baselines by a wide margin.The reported results cover the paper’s evaluated multi-document summarization settings.
- Future Work: Future work will explore more informative graph representations, such as knowledge graphs, to further improve summary quality.This is identified as a direction beyond the current graph representations.
A Appendix
The appendix reports sentence-level ROUGE-L evaluations for the WikiSum and MultiNews test sets, enabling comparison across the two datasets.
- These results are provided so future work can compare model performance across both datasets.
- Sentence-level ROUGE-L is reported for models evaluated on the WikiSum test set.
- Sentence-level ROUGE-L is also reported for models evaluated on the MultiNews test set.