Source-linked AI summary
Neural Document Summarization by Jointly Learning to Score and Select Sentences
Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, Tiejun Zhao
TL;DR
Extractive summarization traditionally separates sentence scoring from sentence selection, creating a gap in jointly modeling importance and redundancy. NEUSUM addresses this with an end-to-end neural model that scores sentences by relative gain given the partial summary, and it significantly outperforms prior extractive methods on CNN/Daily Mail.
Problem
Previous extractive summarization systems treat sentence scoring and sentence selection as separate subtasks.
Method
NEUSUM uses a hierarchical encoder and recurrent sentence extractor to jointly score and select sentences based on relative gain over the partial summary.
Results
NEUSUM achieves 19.01 ROUGE-2 F1 on CNN/Daily Mail and outperforms the strong LEAD3 baseline by 1.31 points.
Takeaways & Limitations
Joint sentence scoring and selection significantly outperforms previous separated methods on the CNN/Daily Mail dataset.
Takeaways & Limitations
Precision drops after successive selections, with error propagation and the greater difficulty of later selections identified as possible reasons.
Abstract
from arXiv · showhide
Sentence scoring and sentence selection are two main steps in extractive document summarization systems. However, previous works treat them as two separated subtasks. In this paper, we present a novel end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences. It first reads the document sentences with a hierarchical encoder to obtain the representation of sentences. Then it builds the output summary by extracting sentences one by one. Different from previous methods, our approach integrates the selection strategy into the scoring model, which directly predicts the relative importance given previously selected sentences. Experiments on the CNN/Daily Mail dataset show that the proposed framework significantly outperforms the state-of-the-art extractive summarization models.
1 Introduction
NEUSUM unifies sentence scoring and selection in an end-to-end neural extractive summarization framework. It models each sentence’s relative gain given the partial summary and extraction state, and reports strong CNN/Daily Mail results.
- 1 Introduction: NEUSUM jointly learns sentence scoring and selection instead of treating them as separate tasks.The model integrates both steps into one end-to-end trainable framework.
- 1 Introduction: The model scores candidates by their relative gain over previously selected sentences.Each decision considers sentence saliency together with the existing partial summary.
- 1 Introduction: A hierarchical document encoder represents sentences, while an RNN extractor tracks the partial summary and extraction state.The extraction state is used to score sentences alongside their representations.
- 1 Introduction: The proposed model is end-to-end trainable without handcrafted features.This is listed as a central model property.
- 1 Introduction: On CNN/Daily Mail, the model significantly outperforms state-of-the-art methods and achieves the best reported result.The experiments compare against separated extractive methods and other baselines.
2 Related Work
Related extractive summarization work models sentence importance and sentence selection through feature-based, graph-based, optimization, and neural sequence-labeling approaches. These methods generally separate scoring from the later extraction decision.
- 2 Related Work: Across these approaches, extractive summarization commonly relies on separate sentence scoring and sentence selection stages.The separation is explicit in the described prior paradigm.
- 2 Related Work: Traditional systems estimate sentence importance with surface features such as term frequency, TF*IDF, length, and position.Feature combinations can be used alone or with learned weights.
- 2 Related Work: Graph-based methods represent sentences as vertices connected by weighted similarity edges, then compute importance with graph algorithms.The resulting vertex scores rank sentences.
- 2 Related Work: MMR selects highly scored sentences while minimizing redundancy with sentences already selected.ILP instead formulates selection as an optimization problem under constraints such as summary length.
- 2 Related Work: Prior neural extractive models include convolutional, attention-based, and sequence-labeling systems.Sequence-labeling approaches classify each sentence for extraction before selecting sentences.
3 Problem Formulation
The formulation treats extractive summarization as selecting a sentence subset under a sentence-count limit while learning summary quality gains. NEUSUM learns the gain associated with adding each candidate after previously selected sentences.
- 3 Problem Formulation: Given document D, the system selects a subset of its sentences to form the output summary.Training uses a reference summary and an evaluation function r(S|S∗).
- 3 Problem Formulation: The output length limit l is defined as a fixed number of sentences.This sentence-count constraint is used in the formulation.
- 3 Problem Formulation: The model adopts MMR because it maximizes relative gain given previously extracted sentences.The learned scoring function is designed around this gain.
- 3 Problem Formulation: ROUGE F1 is used as the evaluation function to avoid favoring longer sentences on CNN/Daily Mail.The dataset does not impose a byte-length summary limit, so the formulation uses a fixed sentence limit.
- 3 Problem Formulation: g(S_t|S_t−1) computes the evaluation-score gain from adding S_t to the previously selected set S_t−1.At each step, the system chooses the sentence with maximal ROUGE F1 gain until reaching the sentence limit.
4 Neural Document Summarization
NEUSUM uses a hierarchical document encoder and a recurrent sentence extractor to jointly score and select sentences while accounting for the partial summary. At each extraction step, it scores remaining sentences using their representations and the current extraction state, then selects one sentence until the output limit is reached.
- Document Encoding: NEUSUM uses a hierarchical document encoder that represents documents through sentence-level and document-level encoding.A BiGRU encodes each sentence, and another BiGRU reads the resulting sentence representations.
- Joint Sentence Scoring and Selection: NEUSUM jointly couples sentence scoring and selection so scoring considers previously selected sentences and selection uses the learned scoring function.This design targets the relative gain over the partial output summary rather than treating scoring and selection as separate tasks.
- Sentence Extraction: The extractor RNN remembers previously selected sentences and produces an extraction state for scoring candidate sentences.The recurrent unit receives the document-level representation of the last extracted sentence, while the scorer uses the resulting hidden state.
- Joint Sentence Scoring and Selection: At each extraction step, the two-layer MLP scores each remaining sentence from the current hidden state and its sentence representation.The model excludes sentences already included in the partial summary when scoring candidates.
- Training and Selection: The model selects the sentence with the maximal gain score and trains its predictions against a smoothed target distribution derived from relative ROUGE F1 gain.Predicted scores are normalized with softmax, while gain values are first rescaled to [0, 1] and then smoothed with a temperature parameter.
5 Experiments
Experiments train and evaluate NEUSUM on the non-anonymized CNN/Daily Mail dataset, comparing it with extractive and abstractive baselines using ROUGE. NEUSUM achieves stronger reported results than the compared extractive systems and is preferred by human evaluators on informativeness, redundancy, and overall quality.
- 5.1 Dataset: The CNN/Daily Mail articles use human-written abstractive highlights, so extractive training labels are constructed by greedily maximizing ROUGE-2 F1.The experiments use the non-anonymized version of the dataset and standard preprocessing.
- 5.2 Implementation Details: At test time, NEUSUM and all baselines extract three sentences for comparability with LEAD3.LEAD3 selects the first three sentences as its summary.
- 5.4 Evaluation: The evaluation reports ROUGE-1, ROUGE-2, and ROUGE-L F1 using the official ROUGE script on the CNN/Daily Mail test set.The reported scores use full-length ROUGE-F1 evaluation, with 95% confidence intervals of at most ±0.22 for the authors’ scores.
- 5.5 Results: 19.01 ROUGE-2 F1 is achieved by NEUSUM, exceeding LEAD3 by 1.31 points and significantly outperforming the neural extractive NN-SE model across ROUGE-1, ROUGE-2, and ROUGE-L F1.The authors also report statistically significant improvements over all baseline models.
- 5.5 Results: Human evaluators rank NEUSUM above NN-SE on informativeness, redundancy, and overall quality, especially for redundancy.Three volunteers evaluated outputs from a random sample of 50 test documents, with ties allowed.
6 Discussion
NEUSUM improves sentence-selection precision by jointly scoring and selecting sentences while conditioning on the partial summary and extraction state. Its selections are closer to the oracle’s position distribution than NN-SE’s, although precision declines across later extraction steps.
- Precision at Step-t: NEUSUM achieves higher extraction precision than NN-SE at every step, including a 3.24 percentage-point advantage at step two.At step two, NEUSUM reaches 31.52% precision versus 28.28% for NN-SE; at step one, it exceeds NN-SE by 1.2 points.
- Precision at Step-t: NEUSUM’s precision drops rapidly after each selection, with p(@1) at 40.38% and p(@2) at 31.52%.The paper attributes the later decline partly to error propagation and the reduced reliability of the selection history.
- Position of Selected Sentences: NN-SE selects about 80.91% of its sentences from the leading three positions, whereas NEUSUM selects 58.64%.NEUSUM also selects more sentences from later ranges, including more than 30% from sentences 4 to 6 and some from sentences 7 to 13.
- Position of Selected Sentences: NEUSUM’s position distribution is closer to the oracle’s than NN-SE’s because it selects more tailing sentences.The paper links this distribution to NEUSUM’s better performance relative to NN-SE.
- Position of Selected Sentences: Joint scoring and selection lets tailing candidates compete directly with leading candidates using the partial summary and current extraction state.The authors suggest this makes NEUSUM more discriminating when a tail sentence is important but only narrowly separated in score from leading sentences.
7 Conclusion
Extractive summarization traditionally separates sentence scoring from sentence selection. The paper combines them in an end-to-end neural framework that conditions scoring on the partial summary and extraction state, and reports significant ROUGE improvements over separated methods.
- 7 Conclusion: NEUSUM combines sentence scoring and sentence selection into one end-to-end phase instead of treating them as separate steps.At each extraction step, it scores sentences according to the partial output summary and current extraction state.
- 7 Conclusion: ROUGE results show that joint sentence scoring and selection significantly outperforms previous separated methods.