Source-linked AI summary
Discourse-Aware Neural Extractive Text Summarization
Jiacheng Xu, Zhe Gan, Yu Cheng, Jingjing Liu
TL;DR
Sentence-based extraction can preserve redundant phrases, while BERT-based encoders have limited ability to model document-wide dependencies. DISCOBERT selects fine-grained EDUs and uses RST and coreference graphs with GCNs to encode their interactions. It achieves new state-of-the-art results on two newswire summarization datasets, outperforming other BERT-base models.
Problem
Sentence-based extraction can retain redundant or uninformative phrases, while BERT-based encoders are limited in capturing long-range document dependencies.
Method
DISCOBERT selects EDUs instead of sentences and applies Graph Convolutional Networks over RST and coreference graphs.
Results
DISCOBERT achieves new state of the art on two popular newswire text summarization datasets, outperforming other BERT-base models.
Takeaways & Limitations
Fine-grained discourse-unit extraction is used to generate more concise and informative summaries with low redundancy.
Takeaways & Limitations
The method adopts RST as its discourse framework and requires restrictions on EDU combinations to ensure grammaticality.
Abstract
from arXiv · showhide
Recently BERT has been adopted for document encoding in state-of-the-art text summarization models. However, sentence-based extractive models often result in redundant or uninformative phrases in the extracted summaries. Also, long-range dependencies throughout a document are not well captured by BERT, which is pre-trained on sentence pairs instead of documents. To address these issues, we present a discourse-aware neural summarization model - DiscoBert. DiscoBert extracts sub-sentential discourse units (instead of sentences) as candidates for extractive selection on a finer granularity. To capture the long-range dependencies among discourse units, structural discourse graphs are constructed based on RST trees and coreference mentions, encoded with Graph Convolutional Networks. Experiments show that the proposed model outperforms state-of-the-art methods by a significant margin on popular summarization benchmarks compared to other BERT-base models.
1 Introduction
DISCOBERT addresses redundancy in sentence-based extraction and weak document-level context modeling by selecting discourse units and encoding discourse graphs. It reports state-of-the-art performance on two newswire summarization datasets.
- BERT and standard encoders remain limited in capturing long-range dependencies throughout documents.
- DISCOBERT selects Elementary Discourse Units as minimal extraction candidates, enabling finer-grained compression and lower redundancy.
- Two discourse graphs, based on RST structure and coreference links, model interactions among discourse units.
- DISCOBERT achieves new state of the art on two popular newswire text summarization datasets, outperforming other BERT-base models.
2 Discourse Graph Construction
The paper constructs RST and coreference graphs over EDUs to represent discourse dependencies and long-range document connections. These structures address fine-grained content selection, grammatical dependencies, and information distributed beyond the document’s beginning.
- RST Graph: EDUs are contiguous, adjacent, non-overlapping text spans tagged as Nucleus or Satellite within RST discourse structure.
- RST Graph: DISCOBERT treats EDUs as minimal selection units so summaries can retain pivotal concepts while removing unnecessary sentence parts.
- RST Graph: EDU selection requires dependency modeling because some individually selected EDUs are not grammatical sentences.
- RST Graph: RST dependency links encode local paragraph-level and long-range document-level connections among EDUs.
- Coreference Graph: Around 25% of oracle sentences appear after the first 10 CNNDM sentences, highlighting information beyond the beginning of documents.
- Coreference Graph: Coreference Graph construction connects discourse units containing mentions from the same coreference cluster.
3 DISCOBERT Model
DISCOBERT encodes documents into EDU representations with BERT, refines them through stacked discourse graph encoders, and predicts EDU-level extractive labels. Its design combines span-level selection with graph-based modeling of discourse relations and long-range dependencies.
- Document Encoder: DISCOBERT first applies BERT to the whole document, then uses a self-attentive span extractor to represent each EDU.EDU representations are weighted sums of BERT hidden states over the corresponding spans.
- EDU-Level Selection: The model selects elementary discourse units rather than whole sentences, formulating extractive summarization as sequential binary labeling over EDUs.During inference, EDUs are ranked by prediction probability while dependency constraints are enforced for grammaticality and coherence.
- Document Encoder: BERT document encoding is adapted for long news articles by inserting sentence-boundary tokens and extending the maximum sequence length from 512 to 768.The text describes articles as typically exceeding 500 words and uses the unshown boundary tokens in the document encoding equations.
- Graph Encoder: Graph Convolutional Networks update EDU representations over graphs whose edges encode RST discourse relations or coreference mentions.Multiple Discourse Graph Encoder layers are stacked, and outputs from different graphs can be concatenated before prediction.
- Training & Inference: Training predicts oracle EDU labels with binary cross-entropy, while oracle construction greedily selects EDUs and inference ranks predicted probabilities subject to EDU dependencies.The graph-enhanced representation is used for prediction when graphs are enabled; the document representation is used in the no-graph variant.
4 Experiments
DISCOBERT is evaluated on CNNDM and NYT against neural summarization baselines, using EDU preprocessing, discourse graphs, and ROUGE evaluation. On CNNDM, the model outperforms sentence-based and competing models, while NYT results show strong gains associated with EDU-level selection.
- Experimental setup: Experiments evaluate neural summarization models on the CNNDM and NYT datasets, comparing DISCOBERT with state-of-the-art baselines.The baselines include extractive, compressive, and BERT-based models.
- Implementation details: Documents are truncated to 768 BPEs, and ROUGE-1, ROUGE-2, and ROUGE-L F1 scores are used for evaluation.R-2 is used as the validation criterion.
- Implementation details: EDU preprocessing uses neural discourse segmentation followed by RST parsing to create the discourse units and structures used by DISCOBERT.The segmenter achieves 94.3 F1 on the RST-DT test set, compared with 98.3 human performance.
- Results on CNNDM: On CNNDM, graph-based DISCOBERT beats the state-of-the-art BERT model by 0.52/0.61/1.04 on R-1/-2/-L on F1.The RST Graph is slightly more helpful than the Coreference Graph, while combining both achieves better overall performance.
- Results on NYT: On NYT, DISCOBERT surpasses the previous state-of-the-art BERT-based model by a significant margin, with most improvement attributed to EDU-level selection.The compared HIBERT model used extra pre-training data.
S and HIBERT∗
DISCOBERT’s evaluation compares discourse-based variants with sentence-based BERT and examines grammaticality, human judgments, and extracted-summary quality. Results indicate gains over BERT in one reported setting, while discourse-graph benefits vary and parsing-related errors remain important.
- Grammaticality: Partial sentence selection can make outputs less grammatical than original sentences, although post-processing rules help complete some EDUs.
- Grammaticality: DISCOBERT summaries retain the quality of the original text according to automatic grammar checking.
- Human Evaluation: Sent-BERT achieves the best overall readability, coherence, and grammaticality, while DISCOBERT is slightly worse but comparable to two other variants.
- Examples & Analysis: DISCOBERT removes a decent amount of irrelevant detail from extracted summaries by selecting discourse units.
- Error Analysis: Errors mostly originate from RST dependency resolution and upstream discourse-parser errors, harming grammaticality and coherence.
5 Related Work
Related work spans neural extractive, compressive, discourse-based, and graph-based summarization. DISCOBERT differs by using EDUs as the selection basis in a neural end-to-end model with discourse graphs.
- Neural Extractive Summarization: Neural extractive summarization has used ranking, index prediction, and sequential labelling for content selection, with BERT-based document encoding also established.
- Neural Compressive Summarization: Prior compressive models use separate selection and compression stages, whereas DISCOBERT achieves sentence compression automatically through EDU-based selection.
- Discourse & Summarization: Earlier discourse-based systems used discourse relations for summarization, trimmed discourse trees, or combined sentence extraction and compression with ILP methods.
- Discourse & Summarization: DISCOBERT is presented as the first neural end-to-end summarization model using EDUs as the selection basis.
- Graph-based Summarization: Graph-based summarization has included stochastic importance estimation, GCNs over relation graphs, graph-based attention, and reasoning over long-distance relationships.
6 Conclusion
DISCOBERT uses discourse units as the minimal selection basis to reduce redundancy and two discourse graphs as inductive bias for long-range dependencies. Across two popular summarization datasets, it shows consistent improvement over baseline models.
- Conclusion: DISCOBERT selects discourse units minimally to reduce summarization redundancy and uses RST and coreference graphs to capture long-range dependencies among them.
- Conclusion: The approach shows consistent improvement over baseline models on two popular summarization datasets.
- Conclusion: Future work will explore better graph encoding methods and apply discourse graphs to tasks requiring long document encoding.
A Appendix
Figure 4 illustrates adjacency matrices for coreference and RST graphs. The appendix text specifies distinct structural properties for the two graph types.
- Appendix: GC is strictly symmetric and includes self-loops on all nodes to prevent the graph from becoming too sparse.
- Appendix: GR has zero diagonal entries because RST nodes do not point to themselves.
- Appendix: Figure 4 presents examples of adjacency matrices for Coreference Graphs GC and RST Graphs GR.