Source-linked AI summary
Graph-based Neural Multi-Document Summarization
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, Dragomir Radev
TL;DR
Multi-document summarization requires modeling relationships among sentences and documents, which prior neural systems often overlook. The paper combines RNN sentence embeddings with GCNs over sentence-relation graphs for salience estimation and redundant-free extraction, outperforming traditional graph-based and no-graph GRU baselines on DUC 2004 while remaining competitive with state-of-the-art MDS systems.
Problem
Prior neural multi-document summarizers process clustered-document sentences independently, overlooking sentence and cross-document relationships, while scaling sequence-to-sequence methods to MDS has not been successful.
Method
The system applies a GCN to sentence-relation graphs using RNN sentence embeddings as node features, then estimates salience and greedily extracts nonredundant sentences.
Results
On DUC 2004, the model outperforms traditional graph-based extractive summarizers and a no-graph GRU baseline, achieving competitive results with other state-of-the-art MDS systems.
Takeaways & Limitations
Combining graph-encoded sentence relations with neural representation learning provides a promising approach for processing multi-document inputs.
Takeaways & Limitations
Selecting sentences solely by highest node degree in Personalized Discourse Graphs did not produce good summaries compared with the GCN model.
Abstract
from arXiv · showhide
We propose a neural multi-document summarization (MDS) system that incorporates sentence relation graphs. We employ a Graph Convolutional Network (GCN) on the relation graphs, with sentence embeddings obtained from Recurrent Neural Networks as input node features. Through multiple layer-wise propagation, the GCN generates high-level hidden sentence features for salience estimation. We then use a greedy heuristic to extract salient sentences while avoiding redundancy. In our experiments on DUC 2004, we consider three types of sentence relation graphs and demonstrate the advantage of combining sentence relations in graphs with the representation power of deep neural networks. Our model improves upon traditional graph-based extractive approaches and the vanilla GRU sequence model with no graph, and it achieves competitive results against other state-of-the-art multi-document summarization systems.
1 Introduction
The paper addresses multi-document summarization systems that ignore relationships among sentences and documents by combining sentence-relation graphs with neural representations. A GCN estimates sentence salience from graph-informed features, enabling redundant-free greedy extraction and competitive DUC 2004 performance.
- Motivation: Neural multi-document summarizers process clustered-document sentences independently, ignoring inter-sentence and cross-document relationships important for MDS.
- Approach: The proposed system applies a GCN to sentence-relation graphs, using RNN sentence embeddings as input node features.The graphs represent sentences as nodes and their relations as edges.
- Approach: Layer-wise graph propagation produces high-level sentence features for salience regression, followed by greedy extraction that avoids redundancy.
- Evaluation: The model is evaluated on the DUC 2004 multi-document summarization task alongside traditional graph-based and neural baselines.
- Results: The model outperforms traditional graph-based extractive summarizers and a no-graph GRU baseline while achieving competitive results with state-of-the-art MDS systems.
2 Related Work
Prior work includes graph-based summarizers using surface, deep, topological, and discourse features, alongside neural systems for compression and single-document summarization. The paper situates its contribution against the limited success of scaling neural sequence-to-sequence methods to MDS.
- Graph-based summarization: Traditional graph-based MDS systems use surface-level, deep-level, topological, and discourse-based sentence relationships to estimate importance.
- Neural summarization: Neural summarization research has applied attention and encoder-decoder architectures to sentence compression, opinion summarization, query-focused summarization, and single-document extraction.
- Neural summarization: Single-document neural systems achieved state-of-the-art results using sequential sentence labeling or copying mechanisms that track source content and summarized material.
- Gap: Scaling neural sequence-to-sequence approaches to multi-document summarization has not been successful, partly because large multi-document summarization datasets are lacking.
3 Method
The method represents sentence relationships as graphs and combines recurrent sentence embeddings with graph convolution to estimate salience for extractive multi-document summarization. It then uses cluster context and greedy selection to produce a nonredundant summary.
- Architecture: The system builds sentence relation graphs, encodes sentences with GRUsent, applies GCN propagation, and selects salient sentences greedily under a length limit.GRUdoc supplies a cluster embedding for salience estimation.
- Graph Representation: Sentence relation graphs connect interacting sentence nodes, using cosine similarity, discourse relations, or personalized discourse information to represent cluster structure.The cosine graph adds edges when tf-idf cosine similarity exceeds 0.2; ADG uses discourse indicators, while PDG personalizes ADG edge weights.
- Graph Representation: PDG enriches ADG by transforming incoming edge weights with sentence personalization scores derived from surface features and normalized across incoming edges.The personalization features are obtained through linear regression and are intended to improve edge-weight diversity and salience predictors.
- Graph Convolutional Networks: GCN takes adjacency matrix A and node features X, then produces high-level node features Z through layer-wise propagation from H0 = X.Self-loops and symmetric degree normalization address omission of a node's own features and changes in feature scale.
- Sentence Embeddings: GRUsent uses the final hidden state of each sentence as its input embedding, and these embeddings form the GCN node-feature matrix.The resulting sentence representations incorporate graph-based sentence relationships before salience estimation.
- Training: Training targets normalized sentence ROUGE scores, where R(si) is based on average ROUGE-1 and ROUGE-2 Recall and sharpened by a validation-selected factor α.The target distribution is normalized across each document cluster with softmax.
4 Experiments
Experiments on DUC 2004 test whether graph representations improve neural multi-document summarization, using four graph/no-graph settings and ROUGE evaluation. The graph-based models outperform the vanilla GRU and traditional graph approaches, with PDG generally strongest and exhibiting favorable training and salience-related properties.
- Experimental Setup: The model is trained on DUC 2001–2002, validated on DUC 2003, and tested on 50 DUC 2004 clusters using ROUGE-1,2.The clusters contain nearly 10 English news documents each, and evaluation uses stemming with stop words retained.
- Experimental Setup: Four experiments compare GRU without graphs against GRU+GCN using Cosine Similarity Graph, ADG, or PDG representations.Graph edges encode sentence relations, with the cosine graph based on tf-idf similarity and the other settings using discourse-oriented graphs.
- Results: PDG improves R-1 by more than 1.50 over the vanilla GRU, while graph-based models outperform the no-graph baseline.ADG performs slightly better than the Cosine Similarity Graph, and PDG improves R-1 by 0.82 over ADG.
- Results: The GCN system significantly outperforms common baselines and traditional graph approaches, exceeds CLASSY04 and SVR, and remains comparable to RegSum.These comparisons use ROUGE-1.5.5 with the same parameters across systems.
- Discussion: PDG converges faster than the no-graph setting and achieves the lowest training and validation costs among the four methods.ADG also has better validation performance than the Cosine Similarity Graph, whereas the no-graph model converges faster but has higher validation cost than graph models.
- Discussion: Node degree positively correlates with sentence salience for all graph types, with correlation strength ordered PDG > ADG > Cosine Similarity Graph.The PDG also connects salient sentences across documents, while simply selecting highest-degree nodes does not match GCN performance.
5 Conclusion
The paper combines neural sentence representations with graph representations of sentence relationships for multi-document summarization. A GCN on a Personalized Discourse Graph captures cross-document relations, improving salience prediction and achieving competitive summarization performance.
- The system combines a GRU-based regression baseline with a GCN applied to a Personalized Discourse Graph.The architecture exploits neural representation power and graph representations of sentence relationships.
- Unlike traditional RNN models, the model captures sentence relations across documents.
- The model demonstrates improved salience prediction and summarization while achieving competitive performance with current state-of-the-art systems.
- Analyses support the efficacy of sentence relation graphs, particularly Personalized Discourse Graphs, for learning sentence salience.