Source-linked AI summary
Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
Yen-Chun Chen, Mohit Bansal
TL;DR
Long-document abstractive summarization is limited by slow encoding and decoding, redundancy, and difficult coordination between sentence selection and rewriting. The paper combines salient-sentence extraction with abstractive rewriting using sentence-level policy gradients, achieving state-of-the-art results, stronger generalization, greater abstractiveness, and substantially faster training and inference. Its conclusion is supported across CNN/Daily Mail and DUC-2002 evaluations.
Problem
Long-document abstractive summarization suffers from slow, inaccurate encoding and decoding, redundancy, and coordination challenges between extraction and rewriting.
Method
The model uses a hybrid extractive-abstractive architecture that selects salient sentences and rewrites them with a sentence-level actor-critic policy-gradient method.
Results
The model achieves state-of-the-art results on CNN/Daily Mail, better generalization on test-only DUC-2002, and substantially faster training and decoding.
Takeaways & Limitations
Sentence-level selection followed by parallel abstractive rewriting combines concise generation with improved quality, fluency, abstractiveness, and computational speed.
Takeaways & Limitations
The feed-forward extractor cannot use the paper’s reinforcement-learning formulation because its historyless structure does not support learning the End-of-Extract action.
Abstract
from arXiv · showhide
Inspired by how humans summarize long documents, we propose an accurate and fast summarization model that first selects salient sentences and then rewrites them abstractively (i.e., compresses and paraphrases) to generate a concise overall summary. We use a novel sentence-level policy gradient method to bridge the non-differentiable computation between these two neural networks in a hierarchical way, while maintaining language fluency. Empirically, we achieve the new state-of-the-art on all metrics (including human evaluation) on the CNN/Daily Mail dataset, as well as significantly higher abstractiveness scores. Moreover, by first operating at the sentence-level and then the word-level, we enable parallel decoding of our neural generative model that results in substantially faster (10-20x) inference speed as well as 4x faster training convergence than previous long-paragraph encoder-decoder models. We also demonstrate the generalization of our model on the test-only DUC-2002 dataset, where we achieve higher scores than a state-of-the-art model.
1 Introduction
The paper combines extractive selection with abstractive rewriting to address long-document encoding, slow decoding, and redundancy. Its hierarchical sentence-level design improves quality while enabling faster parallel processing.
- Abstractive summarizers can be concise but struggle with slow, inaccurate long-document encoding and repetitive multi-sentence outputs.
- The proposed hybrid architecture first selects salient sentences, then compresses and paraphrases them with an abstractor connected through sentence-level policy-gradient reinforcement learning.
- The coarse-to-fine model extracts salient sentences before rewriting them in parallel, improving quality, speed, and stability while reducing redundancy.
- The approach achieves state-of-the-art ROUGE and METEOR results on CNN/Daily Mail, improves generalization on DUC-2002, and produces 3x more novel N-grams than a strong flat model.
- The model’s sentence-level reinforcement learning uses the word-sentence hierarchy, while parallel decoding delivers a reported 10-20x speed-up over the previous best neural abstractive system.
2 Model
The model decomposes summarization into a hierarchical extractor that selects salient document sentences and an abstractor that compresses and paraphrases them. The extractor builds context-aware sentence representations, selects sentences recurrently, and feeds them to an encoder-aligner-decoder abstractor with copying.
- 2 Model: The model approximates latent extraction and abstraction functions by selecting ordered document sentences and rewriting each into a summary sentence.The final summary concatenates the abstractor’s rewritten sentences.
- 2.1 Extractor Agent: A hierarchical encoder computes sentence representations and incorporates context from all previous and future sentences in the document.A temporal convolutional model produces sentence representations, followed by a bidirectional LSTM-RNN.
- 2.1.2 Sentence Selection: A recurrent Pointer Network selects sentences using a decoder with a two-hop attention mechanism over context-aware sentence representations.The decoder attends to sentence representations to obtain a context vector and then attends again to calculate extraction probabilities.
- 2.2 Abstractor Network: The abstractor compresses and paraphrases each extracted sentence with a standard encoder-aligner-decoder and a copy mechanism for out-of-vocabulary words.The abstractor generates the rewritten sentence from the extracted document sentence.
3 Learning
The learning procedure combines proxy supervision, sequence-to-sequence abstractor training, and sentence-level policy-gradient reinforcement learning. Rewards from the rewritten sentences guide extraction, while a stop action and reranking address variable summary length and redundancy.
- 3.2 Reinforce-Guided Extraction: Sentence-level policy gradients connect hard extraction with abstractor rewards in an end-to-end trainable stochastic computation graph.The extractor samples document sentences as actions and receives rewards after the abstractor rewrites them.
- 3 Learning: Proxy extractor labels match each ground-truth summary sentence to its most similar document sentence because datasets lack sentence-level saliency annotations.The extractor is first trained with cross-entropy on these automatically produced labels.
- 3 Learning: The abstractor is trained on extracted document-summary sentence pairs with the usual sequence-to-sequence cross-entropy objective.The objective minimizes the negative log probability of each generated word.
- 3.2 Reinforce-Guided Extraction: A critic baseline reduces vanilla policy-gradient variance, yielding an Advantage Actor-Critic training procedure.The critic predicts state values and is trained with squared loss against the return.
- 3.2 Reinforce-Guided Extraction: The reward uses ROUGE-recall for information coverage during extraction and ROUGE-L F1 for concise sentence rewriting.ROUGE-recall rewards extracted sentences containing information, while ROUGE-L F1 better measures concise abstractor outputs.
- 3.3 Repetition-Avoiding Reranking: A stop action lets the extractor choose how many sentences to produce, while sentence-level trigram reranking reduces repetition across generated sentence candidates.The stop action adapts output length to the document, and reranking favors summaries with fewer repeated n-grams.
4 Related Work
Prior summarization research spans extractive, compressive, and neural abstractive methods, with reinforcement learning and extract-then-generate designs providing related foundations. The paper’s approach is positioned among work on hierarchical attention, coverage, decoding, and scalable extraction.
- 3 Related Work: Earlier summarization methods primarily used extraction or compression, while later neural sequence models advanced abstractive summarization.The related work includes rule-based, statistical, optimization-based, and neural approaches.
- 3 Related Work: Neural abstractive summarization has explored hierarchical attention, coverage, copying, and reinforcement learning to improve generation and training.Prior reinforcement-learning work targeted non-differentiable generation metrics, exposure bias, or extractive selection.
- 3 Related Work: The paper shares high-level intuition with earlier extract-then-compress systems but applies neural extraction and abstraction to summarization.Earlier extract-then-compress work used HMMs, rules, parse trees, and integer linear programming.
- 3 Related Work: Related extract-then-generate systems include machine-translation, question-answering, and scalable parallel extraction models.These systems use fixed networks, sentence or span extraction, synthesis, or cascaded non-recurrent networks.
5 Experimental Setup
Experiments evaluate separate original-text and anonymized CNN/Daily Mail versions, plus a test-only DUC-2002 setting. The study compares modular extractive and abstractive configurations using standard overlap metrics and reinforcement-learning variants.
- 5 Experimental Setup: The experiments train and evaluate separately on original-text and entity-anonymized CNN/Daily Mail versions using the same hyperparameter values.The setup is intended to compare fairly with prior work and test generalization across dataset versions.
- 5 Experimental Setup: The study also evaluates generalization on the out-of-domain, test-only DUC-2002 dataset.DUC-2002 is used without a training split in this evaluation.
- 5 Experimental Setup: Evaluation uses full-length F1 ROUGE-1, ROUGE-2, and ROUGE-L with stemming, plus METEOR.These metrics follow prior summarization evaluations.
- 5.2 Modular Extractive vs. Abstractive: The modular system is tested in extractive and abstractive settings, including feed-forward and recurrent extractors with and without reinforcement learning.The abstractor can be applied to extracted sentences to produce the full abstractive model.
- 5.2 Modular Extractive vs. Abstractive: On original CNN/Daily Mail, reinforcement learning significantly improves all metrics over non-RL recurrent extractors, while reranking significantly improves all ROUGE metrics.Both improvements are reported at p < 0.01.
6 Results
Across CNN/Daily Mail settings and the out-of-domain DUC-2002 test-only setup, the model reports strong improvements, including state-of-the-art results, while also improving efficiency and abstractiveness. Reinforcement learning, reranking, and parallel sentence rewriting contribute to the reported quality and speed results.
- Overall results: The model achieves strong improvements and new state-of-the-art results across extractive and abstractive CNN/Daily Mail settings, with better scores on DUC-2002.The reported comparisons cover original-text and anonymized CNN/Daily Mail versions plus an out-of-domain test-only setup.
- Extractive Summarization: The reinforced extractor performs best because it receives summary-level reward and reduces train-test mismatch from feeding previous extraction decisions.Its improvement over lead-3 is consistent across both CNN/Daily Mail tables, and it outperforms the prior best neural extractive model on the anonymized version.
- Abstractive Summarization: After abstraction, both combined models exceed the pointer-generator baseline without coverage across all metrics.The authors attribute this to extracting multiple sentences with different keypoints, which naturally reduces repetition.
- Abstractive Summarization: After reinforcement learning, the full model significantly outperforms the ML-trained baseline and See et al. (2017) on all metrics for both dataset versions, with p < 0.01.Statistical significance was assessed with a bootstrap test using 100K samples; significance was not tested against Paulus et al. (2018) because its output was unavailable.
- Reranking and Redundancy: Reranking improves ROUGE by removing remaining across-sentence redundancies, while the best model is superior to See et al. (2017) and improves R-L by 0.4 points over Paulus et al. (2018).The model is comparable to Paulus et al. (2018) on R-1 and R-2.
- Speed Comparison: The two-stage hybrid model provides significant train- and test-time speed-ups, including a reported 10-20x test-time advantage over the previous best neural abstractive system.The extractor first computes all selected sentences, after which the abstractor rewrites them concurrently in parallel.
7 Analysis
The model produces more abstractive summaries by rewriting extracted sentences concisely while preserving their facts. An optional reranker further improves global compactness by reducing redundancy across sentences.
- Analysis: The model achieves substantially higher abstractiveness than previous work.Abstractiveness is measured as the ratio of novel n-grams in the generated summary that are absent from the input document.
- Analysis: The abstractor rewrites extracted sentences concisely while keeping the mentioned facts.When rewriting longer text, it has many facts to choose from, creating opportunities for redundancy across sentences.
- Analysis: The optional reranker makes outputs more compact globally by reducing redundancy across sentences.Improved ROUGE scores indicate that reranking successfully removes some remaining redundancies.
8 Conclusion
The paper proposes a sentence-level reinforcement-learning model that captures the word–sentence hierarchy in abstractive summarization. It reports state-of-the-art performance and better generalization together with faster training and decoding.
- Conclusion: The model uses sentence-level reinforcement learning to make abstractive summarization aware of the word–sentence hierarchy.The approach is designed as a sentence-level RL model for abstractive summarization.
- Conclusion: The model achieves state-of-the-art results on both CNN/DM versions and better generalization on test-only DUC-2002.The conclusion also reports a significant speed-up in training and decoding.
- Conclusion: The approach provides a significant speed-up in training and decoding.
Supplementary Materials
The abstractor is a separate attention-based sequence-to-sequence network with copying, while the broader architecture uses sentence representations and decoding mechanisms for abstractive rewriting.
- A.2 Abstractor: The abstractor is a separate sequence-to-sequence network with attention and a copy mechanism, but no coverage.It does not share parameters with the extractor agent.
- A.2 Abstractor: The extractor and abstractor are separate neural submodules in the overall model.The extractor selects salient ordered sentences, which the abstractor rewrites.
- A.2 Abstractor: The sequence-to-sequence model uses a standard encoder-aligner-decoder architecture with bilinear multiplicative attention.The source and target embedding matrix and output projection matrix are shared.
- A.2 Abstractor: The copy mechanism extends decoding to an input-document extended vocabulary.At test time, an out-of-vocabulary prediction is replaced by the document word with the highest attention score.
A.3 Actor-Critic Policy Gradient
The extractor is trained with actor-critic policy gradients because hard sentence selection is non-differentiable. The method samples sentence-selection actions, estimates returns, and uses a critic baseline to reduce variance.
- A.3 Actor-Critic Policy Gradient: REINFORCE estimates the policy gradient from sampled episodes and returns over the sentences extracted by the agent.The Monte Carlo estimate uses the total number of extracted sentences as the episode length.
- A.3 Actor-Critic Policy Gradient: A critic network predicts the state-value baseline, which is subtracted from the action value to estimate the advantage and mitigate variance.The critic shares the pointer-network decoder structure but changes its final layer to regress the state-value function.
- A.3 Actor-Critic Policy Gradient: The average reward rises quickly after the agent selects the EOE action.Before selecting EOE, extra sentence selections receive zero rewards and lower the average.
- A.3 Actor-Critic Policy Gradient: The feed-forward extractor assumes extracted sentences remain in document order and selects the top-k sentences at test time.Its sentence-selection network can be trained with cross-entropy when ground-truth extraction labels are available.
- A.3 Actor-Critic Policy Gradient: The experiments use CNN/Daily Mail in both anonymized and original-text versions, while DUC-2002 is used only for testing transfer.The CNN/Daily Mail split contains 287,227 training, 13,368 validation, and 11,490 test documents; DUC-2002 contains 567 document-summary pairs.
B.3 Training Speed
The model was substantially faster to train than the reported pointer-generator baseline, primarily because its abstractor uses shortened input/target pairs and sentence-pair training.
- 19.71 hours were required to train the model, compared with more than 78 hours reported for See et al. (2017).
- The training-speed gain mainly came from the abstractor’s shortened input/target pairs.
- Sentence-pair encoder-decoder-aligner training was faster than the document-summary training used by the pointer-generator model.
C Generation Samples
The paper illustrates generated summaries on dataset examples, showing extractor-selected sentences and ROUGE scores for a specific example.
- Figures 4 and 5 provide output examples from the dataset for the best models.
- The examples include source documents about a women’s boat race and an unauthorized Full House television movie.
- Colored or marked source sentences indicate the sentences selected by the extractor.
- The reported ROUGE scores are computed for the specific example shown.
- One example reports ROUGE-1: 37.93, ROUGE-2: 17.86, and ROUGE-L: 37.93 for a reranked generated summary.