Source-linked AI summary
Neural Latent Extractive Document Summarization
Xingxing Zhang, Mirella Lapata, Furu Wei, Ming Zhou
TL;DR
Extractive summarization depends on sentence-level labels that are often created heuristically and may be suboptimal. The paper treats sentence selections as latent variables, uses a compression model to infer human summaries from selected sentences, and trains directly against gold summaries. On CNN/Dailymail, the latent model improves over a strong extractive baseline and performs competitively with recent models.
Problem
Extractive summarization commonly requires heuristically generated sentence-level labels, which may be suboptimal when datasets provide only document-summary pairs.
Method
The model treats document sentences as binary latent variables and uses a sentence compression model to infer human summaries from sentences whose variables are activated.
Results
The latent model improves over a strong extractive baseline, while experiments on CNN/Dailymail show competitiveness with several recent models.
Takeaways & Limitations
Training with a loss directly based on gold summaries is useful for improving extractive summarization over heuristic-label training.
Takeaways & Limitations
The compression model is sensitive to normalization, and applying it after extraction can substantially reduce performance because sentence-level compression may remove phrases important for document-level summaries.
Abstract
from arXiv · showhide
Extractive summarization models require sentence-level labels, which are usually created heuristically (e.g., with rule-based methods) given that most summarization datasets only have document-summary pairs. Since these labels might be suboptimal, we propose a latent variable extractive model where sentences are viewed as latent variables and sentences with activated variables are used to infer gold summaries. During training the loss comes \emph{directly} from gold summaries. Experiments on the CNN/Dailymail dataset show that our model improves over a strong extractive baseline trained on heuristically approximated labels and also performs competitively to several recent models.
1 Introduction
Extractive summarization selects source sentences, whereas abstractive methods may generate new words or phrases. Neural approaches use sentence representations, but sentence-level labels can be suboptimal, motivating a latent extractive model trained directly from human summaries.
- Extractive systems select source sentences, while abstractive systems may generate words or phrases absent from the document.
- Prior extractive summarizers framed sentence selection as ranking or binary classification using engineered features, probabilistic models, graphs, optimization, or neural networks.
- Sentence-level True/False labels used to train extractive models may be suboptimal when approximated heuristically.
- The proposed latent extractive model treats sentence labels as binary latent variables and maximizes the likelihood of human summaries given selected sentences.
- On CNN/Dailymail, the latent model improves over a strong extractive baseline trained on rule-based labels and performs competitively with several recent models.
2 Model
The model combines neural extractive summarization with sentence compression, then treats sentence-selection labels as latent variables trained directly against human summaries. It estimates how well selected sentences explain and cover the reference summary, optimizing their expected reward.
- Neural Extractive Summarization: The neural extractive model encodes sentences, contextualizes them with a document-level Bi-LSTM, and predicts a True or False label for each sentence.Sentence representations are produced by averaging word-level Bi-LSTM hidden states; a document encoder and decoder then predict labels.
- Neural Extractive Summarization: The baseline requires sentence-level labels during training, and these labels are obtained heuristically.The model is otherwise trained by minimizing the negative log-likelihood of sentence labels.
- Sentence Compression: The compression model pairs each summary sentence with its most ROUGE-similar document sentence and learns an attention-based sequence-to-sequence mapping.The resulting model estimates the probability that a candidate extracted sentence can be compressed into a target summary sentence.
- Latent Extractive Summarization: The latent model represents each sentence-selection decision as a binary variable and uses selected sentences to infer the human summary.The extractive model samples latent variables, while the compression model estimates summary-sentence likelihoods from the selected sentence set.
- Latent Extractive Summarization: Training minimizes the negative expected reward, combining summary precision and recall, but the expectation is approximated with a single sample because exact computation is exponential.REINFORCE approximates the gradient, using a learned baseline to reduce gradient variance and a pretrained extractive model for initialization.
- Latent Extractive Summarization: Using normalized probabilities is important because experiments with unnormalized probabilities produced inferior results.This comparison concerns the probability normalization used in the latent model's reward calculation.
3 Experiments
Experiments on CNN/Dailymail compare the proposed latent model with extractive, abstractive, and heuristic-label baselines using ROUGE evaluation. The latent model outperforms the strong EXTRACT baseline, while post hoc compression substantially reduces performance.
- Dataset and Evaluation: 287,226 training, 13,368 validation, and 11,490 test document-summary pairs were used from CNN/Dailymail.Sentence labels were created by selecting the sentence subset maximizing ROUGE against each human summary.
- Dataset and Evaluation: Table 1 reports full-length F1 ROUGE-1, ROUGE-2, and ROUGE-L on the CNN/Dailymail test set.The comparison includes LEAD3, extractive systems, and abstractive systems.
- Results: EXTRACT exceeds LEAD3 by +0.75 ROUGE-2 points and +0.57 ROUGE-L points.EXTRACT also outperforms previously published extractive models, although comparison with anonymized SummaRuNNer is not strict.
- Results: EXTRACT outperforms all compared abstractive models except abstract-RL.abstract-RL is more competitive on ROUGE-1 and ROUGE-L, whereas its ROUGE-2 is lower.
- Results: LATENT outperforms the strong EXTRACT baseline, indicating that directly training from gold summaries is useful.The differences among LEAD3, EXTRACT, and LATENT are statistically significant at a 0.95 confidence interval.
- Results: Applying the compression model to LATENT outputs causes a considerable performance drop.The authors suggest sentence-level compression may remove phrases important for document-level summaries.
4 Conclusions
The paper proposes a latent-variable extractive summarization model that uses human summaries directly with sentence compression. Experiments show improvement over a strong extractive model, while applying compression to its outputs produces inferior results.
- Contribution: The proposed latent-variable model leverages human summaries directly with a sentence compression model.The model treats sentence selection variables as latent rather than relying on supplied sentence labels.
- Findings: The latent model improves over a strong extractive model.This is the paper's reported experimental conclusion.
- Limitations and Future Work: Applying the compression model to the extractive system's output leads to inferior output.The authors plan to explore compression models tailored to the summarization task.