Source-linked AI summary
Learning to Extract Coherent Summary via Deep Reinforcement Learning
Yuxiang Wu, Baotian Hu
TL;DR
High-quality summaries from long documents remain challenging because neural abstractive systems struggle with long-document representation and multiple coherent sentences. This paper proposes a neural coherence model and trains RNES with coherence and ROUGE rewards, achieving state-of-the-art ROUGE performance on CNN/Daily Mail while producing more informative and coherent summaries.
Problem
Neural summarization systems struggle with long-document representation and generating multiple coherent sentences, while neural extractive models generally omit coherence from sentence selection.
Method
The paper trains Reinforced Neural Extractive Summarization with a neural coherence model and ROUGE as reinforcement-learning rewards to balance coherence and informativeness.
Results
RNES achieves state-of-the-art ROUGE performance on CNN/Daily Mail, while qualitative evaluation finds its summaries more informative and coherent.
Takeaways & Limitations
The results indicate that reinforcement learning can balance cross-sentence coherence with sentence importance in extractive summarization.
Takeaways & Limitations
The experiments use γ = 1 for simplicity, leaving this modeling assumption fixed in the reported work.
Abstract
from arXiv · showhide
Coherence plays a critical role in producing a high-quality summary from a document. In recent years, neural extractive summarization is becoming increasingly attractive. However, most of them ignore the coherence of summaries when extracting sentences. As an effort towards extracting coherent summaries, we propose a neural coherence model to capture the cross-sentence semantic and syntactic coherence patterns. The proposed neural coherence model obviates the need for feature engineering and can be trained in an end-to-end fashion using unlabeled data. Empirical results show that the proposed neural coherence model can efficiently capture the cross-sentence coherence patterns. Using the combined output of the neural coherence model and ROUGE package as the reward, we design a reinforcement learning method to train a proposed neural extractive summarizer which is named Reinforced Neural Extractive Summarization (RNES) model. The RNES model learns to optimize coherence and informative importance of the summary simultaneously. Experimental results show that the proposed RNES outperforms existing baselines and achieves state-of-the-art performance in term of ROUGE on CNN/Daily Mail dataset. The qualitative evaluation indicates that summaries produced by RNES are more coherent and readable.
Introduction
The paper targets coherent neural extractive summarization because existing neural methods identify important sentences but may produce semantically disconnected summaries. It introduces a neural coherence model and RNES, trained with coherence and ROUGE rewards.
- Neural abstractive summarization remains challenging for long documents and multiple coherent sentences.
- Extractive summarization is more practical because it preserves grammatical correctness and semantic relevance to the document.
- DNN-based extractive methods identify important sentences but may select semantically independent sentences that reduce readability.
- The neural coherence model captures cross-sentence entity transitions and discourse relations using distributed representations, convolution, and max-pooling.
- RNES uses neural coherence outputs as immediate rewards and ROUGE as the final reward to balance coherence with informative importance.
Related Work
Prior extractive and coherence models rely substantially on handcrafted or entity-grid features, while reinforcement-learning summarizers generally optimize other objectives. The paper positions its approach as combining neural extractive summarization, reinforcement learning, and coherence modeling.
- Earlier extractive summarization methods use graph-based methods, submodular functions, integer linear programming, or handcrafted features.
- Reinforcement learning suits extractive summarization because the task makes a sequence of extraction decisions.
- Previous reinforcement-learning summarizers use handcrafted features or ROUGE rewards but do not incorporate coherence into neural extractive summarization.
- Existing coherence models use entity grids or pairwise neural representations, but cited approaches have limitations in feature dependence or cross-sentence local interaction.
Neural Extractive Summarization Model
The NES model represents documents hierarchically, encoding words with convolutional features and sentence context with a bidirectional GRU. It then makes sequential binary extraction decisions using sentence, document, and previously selected-sentence representations.
- Extraction decisions: NES outputs sequential binary decisions, estimating each extraction probability from the current sentence, previously extracted sentences, and the document representation.
- Word-level encoding: NES represents each sentence as a word-embedding matrix and applies multiple convolution kernels to extract word-context features.
- Word-level encoding: The sentence representation is formed from the mean of its word features.
- Sentence-level encoding: A bidirectional GRU models sentence context and concatenates forward and backward hidden states into contextual sentence representations.
- Document encoding: The document representation is obtained through a nonlinear transformation of the mean of sentence representations.
- Training: The model is pretrained with supervised learning by minimizing the negative log-likelihood of ground-truth extraction labels.
Reinforced Neural Extractive Summarization Model
RNES is trained with reinforcement learning to extract summaries by combining neural coherence rewards with ROUGE-based informativeness rewards. Its coherence model uses convolution and max-pooling over sentence pairs to produce scores that guide extraction.
- Reinforcement Learning: RNES treats extractive summarization as sequential decisions, where an agent selects or skips each document sentence.The state contains the document and previous selections, while each action determines whether the current sentence is extracted.
- Reinforcement Learning: REINFORCE updates the RNES policy using sampled states, actions, and their observed returns because exact gradient computation is infeasible in the large state space.The algorithm uses Monte Carlo sampling to approximate the policy gradient.
- Reward Design: The training procedure combines immediate coherence rewards with a final ROUGE reward for the complete extracted summary.The coherence term evaluates adjacent extracted sentences, while ROUGE compares the sampled summary with the human reference.
- Neural Coherence Model: The neural coherence model applies convolution and max-pooling to sentence-pair representations before transforming a fixed-length vector into a coherence score.The architecture models local transitions and progressively higher-level coherence representations across two sentences.
- Neural Coherence Model: The neural coherence model is trained with pairwise ranking so coherent sentence pairs receive higher scores than incoherent pairs.The model uses a large-margin objective for this comparison.
Experiments and Results
Experiments show that the neural coherence model captures cross-sentence coherence, while RNES balances coherence and ROUGE to produce higher-quality summaries on CNN/Daily Mail.
- Neural coherence model: 71.3% accuracy versus 50% for random guessing shows that the neural coherence model captures cross-sentence coherence.The model was tested on approximately 23,000 positive sentence pairs, each with one negative sample.
- Neural coherence model: The coherence model captures coreference, semantic associations, and syntactic patterns across sentences, despite noise from embedded image captions.Examples include links such as “photographer”–“shoot” and patterns such as “As a result” and “According to.”
- RNES training: λ = 0.01 provides a good trade-off because both coherence and ROUGE rewards increase and eventually converge.Higher λ values favor coherence but degrade ROUGE, whereas λ = 0.005 makes coherence rewards approach zero.
- RNES performance: RNES models outperform current state-of-the-art models and NES by a large margin under full-length F1 ROUGE-1, ROUGE-2, and ROUGE-L evaluation.The comparison uses the CNN/Daily Mail test set and reports statistically significant RNES scores against previous best models.
- Qualitative evaluation: A qualitative example shows that coherence reward changes extraction order, adding an antecedent before a pronoun and producing a more coherent, readable summary.The coherence model gives a higher score when the sentence pair forms a coherent sequence, strengthening the earlier extraction action during REINFORCE training.
Conclusion
The paper concludes that RNES extracts summaries that balance cross-sentence coherence with sentence importance and achieves state-of-the-art performance on the benchmark dataset.
- Conclusion: RNES extracts coherent and informative summaries from a single document while balancing cross-sentence coherence and sentence importance.The authors identify improving the neural coherence model and introducing human knowledge as future work.