Source-linked AI summary
Heterogeneous Graph Neural Networks for Extractive Document Summarization
Danqing Wang, Pengfei Liu, Yining Zheng, Xipeng Qiu, Xuanjing Huang
TL;DR
Extractive summarization needs effective modeling of cross-sentence relations, while existing approaches have limitations in capturing long-distance dependencies and graph structure. The paper proposes HeterSumGraph, which adds semantic nodes as sentence intermediaries and extends naturally to multi-document settings through document nodes. The model outperforms existing competitors on three benchmark datasets without pretrained language models and achieves the best CNN/DailyMail results among non-BERT models.
Problem
Existing summarization models face challenges in effectively modeling cross-sentence relations, especially long-distance dependencies in long or multi-document inputs.
Method
HeterSumGraph uses a heterogeneous graph containing sentence nodes and additional semantic-unit nodes that mediate sentence interactions, with document nodes enabling multi-document extension.
Results
The model outperforms all existing competitors on three benchmark datasets without pretrained language models and achieves the best CNN/DailyMail results among non-BERT models.
Takeaways & Limitations
Introducing fine-grained semantic units helps build more complex sentence relationships, while the graph framework remains adaptable to multi-document summarization.
Takeaways & Limitations
The experiments exclude pretrained contextualized encoders because of limited computational resources, leaving their integration for future work.
Abstract
from arXiv · showhide
As a crucial step in extractive document summarization, learning cross-sentence relations has been explored by a plethora of approaches. An intuitive way is to put them in the graph-based neural network, which has a more complex structure for capturing inter-sentence relationships. In this paper, we present a heterogeneous graph-based neural network for extractive summarization (HeterSumGraph), which contains semantic nodes of different granularity levels apart from sentences. These additional nodes act as the intermediary between sentences and enrich the cross-sentence relations. Besides, our graph structure is flexible in natural extension from a single-document setting to multi-document via introducing document nodes. To our knowledge, we are the first one to introduce different types of nodes into graph-based neural networks for extractive document summarization and perform a comprehensive qualitative analysis to investigate their benefits. The code will be released on Github
1 Introduction
Extractive summarization requires modeling cross-sentence relations, but recurrent models struggle with long-distance dependencies and effective graph structures are difficult to design. HeterSumGraph addresses this with heterogeneous semantic nodes that mediate sentence interactions and supports extension to multi-document summarization.
- RNN-based models often struggle to capture sentence-level long-distance dependencies in long or multi-document inputs.
- HeterSumGraph adds semantic-unit nodes alongside sentence nodes to model cross-sentence relations in a heterogeneous graph.These additional nodes act as intermediaries connecting sentences during message passing.
- Word nodes connect to their containing sentences, enabling interaction through explicit overlapping-word information without direct edges between all sentence pairs.Word nodes also aggregate information from sentences and are updated during graph propagation.
- Document nodes provide a direct extension of the heterogeneous graph from single-document to multi-document summarization.
- On three benchmark datasets, the model outperforms existing competitors without pretrained language models.
2 Related Work
Prior extractive summarization methods commonly use encoder-decoder architectures or homogeneous graphs with one node type. Heterogeneous graph methods instead represent multiple semantic node types and motivate their use for richer node representations in summarization.
- Neural extractive summarization has primarily used recurrent or Transformer-based encoders within encoder-decoder frameworks.
- Graph-based summarization models use statistical or linguistic sentence relations, including content similarity and relational priors.
- Existing graph neural approaches generally formulate documents as homogeneous graphs containing only one node type.
- Heterogeneous graph research models graphs with multiple node types using methods such as message passing and self-attention.
- Prior summarization systems used heterogeneous graphs with topic, word, sentence, or keyword nodes, while this work applies heterogeneous graph neural networks to extractive summarization.
3 Methodology
The methodology represents documents as heterogeneous word–sentence graphs, updates both node types through iterative GAT message passing, and classifies sentence nodes for extraction. Document supernodes extend the same framework to multi-document summarization.
- Task Formulation: Extractive summarization is formulated as sequence labeling, predicting binary inclusion labels for document sentences.Ground-truth ORACLE labels are generated greedily using ROUGE.
- Heterogeneous Graph Construction: The graph contains word relay nodes and sentence or document supernodes, allowing high-level discourse units to connect through shared semantic nodes.HETERSUMGRAPH uses sentences as supernodes, while HETERDOCSUMGRAPH additionally introduces document nodes.
- Graph Initializers: Graph initializers encode sentence nodes by concatenating CNN-derived local n-gram features with BiLSTM-derived global features.Word embeddings initialize word nodes, while TF-IDF values are infused into word–sentence edge weights.
- Heterogeneous Graph Layer: The heterogeneous graph layer iteratively passes messages from words to sentences and then from updated sentences back to words using GAT-based updates.The layer uses attention weights, residual connections, edge features, and position-wise feed-forward transformations.
- Sentence Selection: Updated sentence representations are classified to predict summary labels, with trigram blocking applied during decoding to reduce overlap.Sentences are ranked by scores, and candidates sharing trigrams with predecessors are discarded.
- Multi-Document Extension: Adding document supernodes adapts the framework to multi-document summarization, with word nodes bridging sentence and document relationships.Documents connect through shared word-mediated content, while sentences containing common words connect across documents.
4 Experiment
The experiments evaluate the proposed extractive summarization models on single- and multi-document datasets, using specified preprocessing, architecture, training, and decoding settings. Comparisons include recurrent and Transformer-based extractive baselines, while HETERSUMGRAPH directly selects summary sentences by node classification.
- Datasets: The evaluation covers CNN/DailyMail and NYT50 for single-document summarization, and Multi-News for multi-document summarization.Multi-News examples contain 2–10 source documents; its inputs are truncated to 500 tokens.
- Implementation: The datasets use fixed vocabulary, embedding, node, graph, sequence-length, and hidden-size settings for both summarization settings.The vocabulary is limited to 50,000 tokens with 300-dimensional GloVe embeddings; documents are truncated to 50 sentences.
- Implementation: Training uses batch size 32, Adam with learning rate 5e-4, early stopping after three validation epochs without loss descent, and one graph iteration.Decoding selects top-3 sentences for CNN/DailyMail and NYT50, and top-9 for Multi-News.
- Models for Comparison: The comparison includes an Ext-BiLSTM sequence encoder and an Ext-Transformer sentence-level fully connected graph baseline.Ext-BiLSTM also serves as an ablation of sentence-node updating in HETERSUMGRAPH.
- Models for Comparison: HETERSUMGRAPH models sentence relations through common words and selects summary sentences by node classification, with trigram blocking available to reduce redundancy.The paper also reports a CNN/DailyMail comparison table using Rouge against recently released summarization systems.
5 Results and Analysis
Across CNN/DailyMail, NYT50, and Multi-News, HeterSumGraph generally outperforms prior non-BERT-based systems by modeling cross-sentence relations with heterogeneous graphs. Ablations and qualitative analyses further examine the effects of graph components, document nodes, trigram blocking, and word-node degree.
- CNN/DailyMail: HeterSumGraph improves over Ext-BiLSTM by more than 0.6/0.51/0.7 on R-1, R-2 and R-L, respectively, on CNN/DailyMail.The authors attribute this to sentence-word-sentence cross-sentence relationship modeling and report further gains over fully connected Ext-Transformer.
- CNN/DailyMail: HeterSumGraph outperforms previous non-BERT-based summarization systems, while trigram blocking greatly improves all ROUGE metrics on CNN/DailyMail.Compared with HER without policy gradients, HeterSumGraph improves R-1 by 0.61, and its trigram-blocked version improves reorganized summaries by 0.65.
- NYT50: On NYT50, the proposed models again outperform BiLSTM, Transformer, and other non-BERT-based approaches, although trigram blocking is less effective than on CNN/DailyMail.The authors attribute this difference to the way summaries are written for the dataset.
- Ablation: Removing low-TF-IDF-word filtering raises R-1 and R-L but lowers R-2, while residual connections, edge features, word updates, and BiLSTM initialization are effective components.The authors suggest filtering noisy words improves focus on useful word nodes but sacrifices some bigram information.
- Multi-News: Both HeterSumGraph and HeterDocSumGraph outperform previous methods on Multi-News, with document nodes producing larger performance improvements.The authors associate this advantage with modeling document-document relationships; trigram blocking does not work well for Multi-News.
- Qualitative Analysis: HeterSumGraph performs much better on documents with higher average word-node degree, supporting the role of sentence-information aggregation and global-representation propagation.Average word-node degree measures word occurrence across sentences and relates to document redundancy.
6 Conclusion
The paper proposes a heterogeneous graph-based neural network for extractive summarization, using fine-grained semantic units and document nodes to model complex sentence relationships across single- and multi-document settings.
- The model introduces fine-grained semantic units into the summarization graph to build more complex relationships between sentences.
- Document nodes make the single-document graph conveniently adaptable to multi-document summarization.
- The models achieve the best results on CNN/DailyMail among non-BERT-based models.
A Appendices
The appendix selects one iterative update for HeterSumGraph because three iterations provide comparable validation performance while nearly doubling epoch time.
- HeterSumGraph has comparable results with t = 1 and t = 3.
- Increasing the iteration number from 1 to 3 nearly doubles the time for one epoch, so the authors select t = 1.The choice balances time cost and model performance.