Source-linked AI summary
Sentence Centrality Revisited for Unsupervised Summarization
Hao Zheng, Mirella Lapata
TL;DR
Large-scale, high-quality training data may not be available for every summarization style, domain, or language, motivating an unsupervised alternative. The paper revises graph-based centrality with BERT sentence representations and position-sensitive directed edges, outperforming strong baselines across three news datasets. Its best system is comparable to supervised systems trained on hundreds of thousands of examples.
Problem
Large-scale and high-quality training data may not be available for different summarization styles, domains, or languages.
Method
The paper revisits graph-based ranking by using BERT to represent sentences and directed edges to incorporate their relative document positions.
Results
The approach outperforms strong baselines across three news summarization datasets, while its best system is comparable to supervised systems trained on hundreds of thousands of examples.
Takeaways & Limitations
Directed centrality improves salient-content selection, and pretrained embeddings show promise for unsupervised summarization.
Abstract
from arXiv · showhide
Single document summarization has enjoyed renewed interests in recent years thanks to the popularity of neural network models and the availability of large-scale datasets. In this paper we develop an unsupervised approach arguing that it is unrealistic to expect large-scale and high-quality training data to be available or created for different types of summaries, domains, or languages. We revisit a popular graph-based ranking algorithm and modify how node (aka sentence) centrality is computed in two ways: (a)~we employ BERT, a state-of-the-art neural representation learning model to better capture sentential meaning and (b)~we build graphs with directed edges arguing that the contribution of any two nodes to their respective centrality is influenced by their relative position in a document. Experimental results on three news summarization datasets representative of different languages and writing styles show that our approach outperforms strong baselines by a wide margin.
1 Introduction
The paper targets the limited availability of high-quality training data across summary types, domains, and languages by improving unsupervised graph-based summarization with BERT representations and directed edges. Across three news datasets, the approach outperforms strong baselines and approaches supervised systems trained on hundreds of thousands of examples.
- High-quality large-scale training data may not be available across different summarization styles, domains, or languages, motivating unsupervised methods.
- The method improves centrality computation using BERT for sentence similarity and directed edges weighted according to relative sentence position.
- Directed centrality addresses cases where a commenting sentence depends on a preceding event sentence whose content is more central.
- The approach is evaluated on three news summarization datasets spanning different languages, writing conventions, and summary styles.
- The best system is comparable to supervised systems trained on hundreds of thousands of examples and outperforms the leading-sentence baseline.
- The results indicate that directed centrality substantially improves salient-content selection and highlight pretrained embeddings as promising for unsupervised summarization.
2 Centrality-based Summarization
The centrality-based framework represents sentences as similarity-weighted graph nodes, ranks them by centrality, and selects the highest-ranked sentences. The paper motivates directed graphs through discourse salience and approximates nuclearity using relative sentence position.
- Graph-based summarization: Graph-based summarization represents sentences as nodes, weights edges by sentence similarity, and uses degree or PageRank centrality to measure salience.
- Graph-based summarization: After centrality scoring, sentences are sorted in reverse order and the top-ranked sentences are included in the summary.
- Directed Text Graph: Directed edges encode unequal contributions between sentences based on forward- and backward-looking relationships.
- Directed Text Graph: RST motivates directionality by distinguishing central nuclei from peripheral satellites in discourse structure.
- Directed Text Graph: Relative position approximates discourse nuclearity because discourse parsers require annotated corpora and broader NLP tools that may not exist across settings.
- Directed Text Graph: The directed centrality formula uses separate λ1 and λ2 weights for forward- and backward-looking edges, with equal weights reducing it to degree centrality.
3 Sentence Similarity Computation
The approach uses BERT to encode sentences and fine-tunes it with a sentence-level distributional objective before computing and normalizing pairwise similarities.
- BERT as Sentence Encoder: BERT maps sentences into deep continuous representations using a bidirectional Transformer encoder pretrained with masked language modeling and next sentence prediction.The resulting representations are used to encode sentences for unsupervised summarization.
- Sentence-level Distributional Hypothesis: Sentence-level fine-tuning treats each sentence’s neighboring sentences as positive examples and other corpus sentences as negatives.Negative sampling makes the objective more computationally efficient than reconstructing surrounding sentences.
- Sentence-level Distributional Hypothesis: The fine-tuning objective distinguishes context sentences from other corpus sentences, encouraging the encoder to capture the intended sentence’s meaning.The method samples five negative examples for each positive example to approximate the expectation.
- Similarity Matrix: Pairwise dot products produce an unnormalized sentence-similarity matrix, which the method uses instead of cosine similarity based on empirical performance.The representations are obtained for all sentences in a document before pairwise similarity computation.
- Similarity Matrix: The final similarity matrix emphasizes relative similarity contributions, then sets scores below the β threshold to zero.This normalization reduces the effect of absolute values when representations assign high scores to many sentence pairs.
4 Experimental Setup
The experiments evaluate PACSUM on three news summarization datasets spanning languages, information distributions, and summary styles, using dataset-specific preprocessing and tuning procedures.
- Experimental Setup: PACSUM is the shorthand for Position-Augmented Centrality based Summarization, the unsupervised approach evaluated in this section.The section presents the experimental setup for evaluating the proposed summarization method.
- Datasets: The evaluation uses three datasets representing different languages, document information distributions, and summary styles.The datasets are NYT, CNN/DailyMail, and TTNews.
- CNN/DailyMail: CNN/DailyMail uses news articles with associated highlights, and its standard training, validation, and test splits are retained without entity anonymization.The splits contain 90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents.
- NYT and CNN/DailyMail: NYT complements CNN/DailyMail because salient information is distributed more evenly through its articles, whereas CNN/DailyMail strongly favors leading sentences.LEAD-3 is described as extremely difficult to beat on CNN/DailyMail.
- TTNews: TTNews tests applicability across languages with concise Chinese summaries shaped by limited mobile-phone display space.The corpus contains news articles and corresponding human-written summaries from the Toutiao app.
- Training and Evaluation: BERT is fine-tuned separately for each dataset using training documents, while hyperparameters are tuned on 1,000 validation examples with gold summaries.English and Chinese BERT versions are used for the corresponding corpora, with five negative examples sampled per positive example.
5 Results
PACSUM is evaluated against unsupervised and supervised baselines on English and Chinese news summarization using ROUGE and question-answering judgments. Across datasets and evaluation settings, directed centrality with BERT representations performs strongly, while position weighting substantially affects results.
- Automatic evaluation: ROUGE F1 evaluates informativeness with unigram and bigram overlap and fluency with longest common subsequence.
- NYT and CNN/Daily Mail: PACSUM with BERT representations achieves the highest ROUGE F1 among unsupervised approaches on NYT and CNN/Daily Mail.The comparison includes TextRank variants using tf-idf, skip-thought, and BERT representations, plus degree centrality with tf-idf.
- NYT and CNN/Daily Mail: Approximately 6 absolute ROUGE-1 F1 points separate PACSUM with BERT from LEAD-3 on NYT.Despite using only 1,000 validation examples for hyper-parameter tuning, the best system is comparable to supervised systems trained on hundreds of thousands of examples.
- NYT and CNN/Daily Mail: PACSUM substantially outperforms TextRank across sentence representations, supporting the benefit of position information for determining centrality.Degree centrality with tf-idf is very close to TextRank with tf-idf on NYT and CNN/Daily Mail.
- Hyper-parameter analysis: On the NYT validation set, PACSUM’s ROUGE-1 F1 ranges from 0.30 to 0.40 as λ1 varies from -2 to 1, with λ2 = 1.The optimal λ1 is negative, suggesting that similarity with previous content hurts centrality in this case.
- TTNews: PACSUM outperforms LEAD and TextRank on the Chinese TTNews corpus, despite its typically one-sentence summaries disadvantaging extractive methods.Pointer-generator is superior to unsupervised methods and comes close to the extractive oracle on TTNews.
- Human evaluation: In QA evaluation, PACSUM significantly outperforms LEAD but remains below the oracle and performs on par with REFRESH.The oracle’s sub-100 performance indicates that ROUGE-maximizing extraction can miss salient content and capture surface similarity instead.
6 Conclusions
The paper presents an unsupervised summarization system with modest data requirements and portability across summary types, domains, and languages. It refines graph-based sentence centrality using BERT and directed edges, outperforming strong baselines on three news datasets.
- The system combines BERT-based sentence similarity with directed graph edges whose centrality contributions depend on document position.The approach revisits graph-based ranking and modifies sentence centrality computation.
- Experiments on three news summarization datasets demonstrate superiority over strong baselines.
- The unsupervised system requires modest data and is portable across summary types, domains, and languages.
A.1 Examples of System Output
Table 6 compares gold summaries with outputs from LEAD, TextRank, and PACSUM across test documents from NYT, CNN/Daily Mail, and TTNews.
- Table 6 shows examples of system output for test documents from NYT, CNN/Daily Mail, and TTNews.
- The examples compare gold summaries with summaries produced by LEAD, TextRank, and PACSUM.GOLD is the reference summary; LEAD extracts the first document sentences.
- TextRank computes sentence centrality recursively using PageRank and a Markov chain model.
NYT CNN+DM TTNews
Table 6 presents example gold summaries and system outputs for documents from NYT, CNN/Daily Mail, and TTNews.
- The table presents example gold summaries and system outputs.
- Examples are provided for NYT documents.
- Examples are also provided for CNN/Daily Mail and TTNews documents.