Source-linked AI summary
Multi-document abstractive summarization using ILP based multi-sentence compression
Siddhartha Banerjee, Prasenjit Mitra, Kazunari Sugiyama
TL;DR
Abstractive summarization seeks to synthesize information from multiple documents into concise summaries, but prior text-to-text methods did not explicitly model linguistic quality. This paper clusters aligned sentences, generates word-graph paths, and selects them with an ILP optimizing information and readability; on DUC 2004 and 2005, it outperforms baselines and state-of-the-art extractive systems.
Problem
Prior text-to-text methods did not explicitly model linguistic quality, although abstractive summaries should be coherent, readable, and grammatically correct.
Source: §1 · p. 1Method
The method clusters aligned sentences from multiple documents, generates word-graph paths, and uses ILP to select paths by information and linguistic-quality scores.
Source: §1 · p. 1Results
On DUC 2004 and 2005, the method outperforms baselines and state-of-the-art extractive summarizers by ROUGE, and human judges rate its linguistic quality above MSC and informativeness above DPP and MSC.
Source: §1 · p. 2, §4.3 · p. 6, §5 · p. 7Takeaways & Limitations
The approach aggregates information from multiple sentences while using an inter-sentence constraint to avoid redundant content across clusters.
Source: §1 · p. 1The word-graph’s adjacency-based language modeling can produce contextually incoherent phrases; the authors propose dependency parses as future work.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Abstractive summarization is an ideal form of summarization since it can synthesize information from multiple documents to create concise informative summaries. In this work, we aim at developing an abstractive summarizer. First, our proposed approach identifies the most important document in the multi-document set. The sentences in the most important document are aligned to sentences in other documents to generate clusters of similar sentences. Second, we generate K-shortest paths from the sentences in each cluster using a word-graph structure. Finally, we select sentences from the set of shortest paths generated from all the clusters employing a novel integer linear programming (ILP) model with the objective of maximizing information content and readability of the final summary. Our ILP model represents the shortest paths as binary variables and considers the length of the path, information score and linguistic quality score in the objective function. Experimental results on the DUC 2004 and 2005 multi-document summarization datasets show that our proposed approach outperforms all the baselines and state-of-the-art extractive summarizers as measured by the ROUGE scores. Our method also outperforms a recent abstractive summarization technique. In manual evaluation, our approach also achieves promising results on informativeness and readability.
1 Introduction
The approach combines information across documents while jointly optimizing summary informativeness and readability, addressing linguistic quality and redundancy. On DUC 2004 and 2005, it outperforms baselines and state-of-the-art extractive systems, with manual evaluations also reporting favorable quality.
1 Introduction
- Because prior text-to-text methods did not explicitly model linguistic quality, the approach uses a language-model log probability and jointly maximizes readability and information content.It also aggregates information from multiple sentences and constrains redundancy across clusters, addressing information loss and repeated content associated with extractive summarization and MSC.Source: §1 · p. 1
- The system selects an important document, clusters similar sentences across documents, generates K-shortest paths in word graphs, and uses ILP to select paths for the summary.The objective combines path information and linguistic-quality scores, while constraints limit selection to one sentence per cluster and reduce inter-cluster redundancy.Source: §1 · pp. 1–2
- On DUC 2004 and 2005, the method outperforms all baselines and state-of-the-art extractive summarizers by ROUGE, and also beats an MSC-based abstractive system on ROUGE-2, ROUGE-L, and ROUGE-SU4.Human judges also found its summaries linguistically acceptable and highly informative.Source: §1 · p. 2
2 Related Work
Prior abstractive summarization work included NLG systems, induced templates, supervised compression, and graph-based approaches. The paper distinguishes its unsupervised, template-free method and joint optimization of informativeness and readability from these approaches.
2 Related Work
- NLG systems require manual schema definition and deeper language analysis, while induced templates are described as ineffective for variable-topic news summarization.The authors contrast these approaches with their unsupervised method, which combines information from same-topic sentences without inducing templates.Source: §2 · p. 2
- A prior ILP approach jointly extracts and compresses sentences but is supervised and requires substantial manual effort to define subtree-deletion features.The paper contrasts that setup with its own unsupervised summarizer.Source: §2 · p. 2
- Mehdad et al. use a supervised entailment graph for meeting summaries and MSC for sentence fusion; Filippova’s ranking relies on word-collocation edge weights and yields low linguistic quality.The paper says its own selection jointly maximizes informativeness and readability.Source: §2 · p. 2
3 Proposed Approach
The approach builds clusters around sentences in a collection’s most important document, generates candidate paths from each cluster, and uses an ILP to select informative, readable, nonredundant summary sentences.
3 Proposed Approach
- The full pipeline generates similar-sentence clusters from a document set, creates word graphs for them, and selects at most one novel sentence per graph for the summary.The selection objective targets information content and linguistic quality across the summary.Source: §3 · p. 2
S1: Sentence Clustering
- Sentence clusters start from the most important document; sentences from other documents are aligned by cosine similarity, with alignments below 0.5 excluded and clusters retained only when they cover at least half the documents.The paper orders clusters using majority ordering or average normalized sentence position.Source: S1: Sentence Clustering · p. 3, (S1-2) Cluster Generation · p. 3
- The method estimates document importance with document-graph LexRank, average pairwise cosine similarity, or similarity to the concatenated document set.For LexRank, the highest-scoring document is selected as Dimp.Source: (S1-1) Document Importance · p. 3
S2: Summary Sentence Generation
- Word-graph paths combine sentence material, and the system removes paths too similar to source sentences, enforces an eight-word minimum, and retains at most 200 paths per cluster.The context of words is considered when multiple graph mappings are possible.Source: S2: Summary Sentence Generation · pp. 3–4, Figure 2 · p. 4
- Candidate paths are scored for informativeness by summed TextRank word scores and for readability by a smoothed trigram language-model score.The language-model score uses conditional probabilities for word trigrams.Source: Best Path Selection · p. 4
- The ILP selects binary path variables to maximize informativeness and linguistic quality while favoring shorter paths, limiting each cluster to one path, and blocking cross-cluster redundancy at cosine similarity ≥0.5.The path-length term favors shorter informative sentences.Source: ILP Formulation · pp. 4–5
4 Experimental Results
Across DUC 2004 and 2005, the proposed systems outperform the baselines on ROUGE, with the best variant outperforming the other techniques. Human ratings and parser scores also favor ILPSumm on several informativeness and readability measures, though grammaticality errors remain.
4.1 Dataset and Evaluation Metrics
- On DUC 2004 and 2005, summaries are evaluated with ROUGE against human-written model summaries.The manual evaluation asks judges to rate informativeness and linguistic quality on a 1–5 scale.Source: §4.1 · p. 5, §4.3 · p. 6
4.2 ROUGE Evaluation
- All proposed systems outperform the baselines on every reported ROUGE score across both datasets, and ILPSumm is statistically significant against all baselines.The best-performing configuration uses MDImp and majority ordering; ILPSumm also outperforms MSC while content selection is at par with the best extractive systems.Source: §4.2 · pp. 5–6
4.3 Manual Evaluation
- ILPSumm scored 3.63 for linguistic quality versus MSC’s 2.83, and 3.90 for informativeness versus DPP’s 3.78 and MSC’s 3.78.Parser Avg.LL was -180.76 for ILPSumm and -210.02 for MSC; the paper treats the lower magnitude as better.Source: §4.3 · p. 6, Table 2 · p. 6
- The authors report remaining linguistic-quality errors, including incoherent repeated wording and contextually awkward phrases.A language model may miss a repeated word when adjacent n-grams are individually grammatical; future work proposes dependency parses rather than adjacency relations for word-graph construction.Source: §4.3 · pp. 6–7
5 Conclusions and Future Work
The paper clusters redundant information and uses ILP to choose word-graph paths for summaries optimized for information content and linguistic quality. On DUC 2004 and 2005, it reports gains over baselines and state-of-the-art extractive systems, while human judgments favor its summaries over the baseline abstractive system.
5 Conclusions and Future Work
- On DUC 2004 and 2005, the approach outperforms all baselines and state-of-the-art extractive summarizers.Human judgments also find its summaries linguistically preferable to the baseline abstractive summarization technique.Source: §5 · p. 7
- The method clusters redundant information and uses an ILP to select word-graph paths that maximize information content and linguistic quality.Source: §5 · p. 7
- Human judgments find the abstractive summaries linguistically preferable to the baseline abstractive summarization technique.Source: §5 · p. 7
- Future work will use paraphrasing to enhance summary quality and address phrase-level redundancies to improve coherence.Source: §5 · p. 7