Source-linked AI summary
BanditSum: Extractive Summarization as a Contextual Bandit
Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, Jackie Chi Kit Cheung
TL;DR
Extractive summarization needs training methods that avoid heuristic sentence labels and the limitations of sequential selection. BANDITSUM treats summarization as a contextual bandit and uses policy-gradient reinforcement learning to select sentences by ROUGE reward. It performs better than or comparably to state-of-the-art extractive systems, converges with fewer update steps, and handles late summary-worthy sentences better.
Problem
Existing extractive summarizers rely on sentence-level labels or sequential reinforcement-learning decisions that create difficult exploration and favor earlier sentences.
Method
BANDITSUM models each document as a contextual-bandit context and each ordered subset of sentences as an action, training neural networks with policy-gradient reinforcement learning to optimize ROUGE.
Results
BANDITSUM performs better than or comparably to state-of-the-art extractive models, converges with significantly fewer update steps, and performs better when good sentences appear late.
Takeaways & Limitations
Contextual-bandit training provides a promising extractive-summarization framework that improves exploration of sentence selections, particularly when summary-worthy sentences occur late.
Takeaways & Limitations
Reported results are not directly comparable to results based on the anonymized dataset used by Nallapati et al. (2017).
Abstract
from arXiv · showhide
In this work, we propose a novel method for training neural networks to perform single-document extractive summarization without heuristically-generated extractive labels. We call our approach BanditSum as it treats extractive summarization as a contextual bandit (CB) problem, where the model receives a document to summarize (the context), and chooses a sequence of sentences to include in the summary (the action). A policy gradient reinforcement learning algorithm is used to train the model to select sequences of sentences that maximize ROUGE score. We perform a series of experiments demonstrating that BanditSum is able to achieve ROUGE scores that are better than or comparable to the state-of-the-art for extractive summarization, and converges using significantly fewer update steps than competing approaches. In addition, we show empirically that BanditSum performs significantly better than competing approaches when good summary sentences appear late in the source document.
1 Introduction
BANDITSUM frames extractive summarization as a contextual bandit, replacing sequential binary labeling with reinforcement-learning training that directly targets summary quality. Experiments report competitive performance, faster convergence, and improved handling of summary-worthy sentences appearing late in documents.
- Extractive methods select and copy document text, offering simpler, faster, and more reliably grammatical summaries than abstractive generation.
- Sequential binary-labeling approaches face a large reinforcement-learning search space and bias selection toward earlier sentences.For document length T, the extractive action space contains 2^T possible summaries.
- BANDITSUM formulates extractive summarization as a contextual bandit and trains neural summarizers with reinforcement learning without supervised pre-training.The method takes a document as input, produces sentence affinities, and samples sentences without replacement.
- BANDITSUM removes systematic preference for earlier sentences while retaining performance when good sentences occur early and improving performance when they occur late.
- BANDITSUM achieves state-of-the-art performance on multiple datasets and requires significantly fewer update steps than competing approaches.
- Human judges rate BANDITSUM summaries as less redundant and higher in overall quality than summaries from competing approaches.
2 Related Work
Related work spans classical and neural extractive summarization, including recurrent, convolutional, and hybrid architectures. Reinforcement-learning approaches have addressed sentence selection, but prior methods faced computational, function-approximation, or performance limitations.
- Neural network-based methods have recently gained popularity over classical approaches to extractive summarization.
- Neural summarization systems use word embeddings and document representations that feed into decoder networks to generate summaries.
- Earlier reinforcement-learning methods used TD(λ) to learn value functions for sentence selection.
- Shallow function approximators limited earlier reinforcement-learning methods, requiring fresh reinforcement learning for every new document.
- Deep Q-learning for extractive summarization required a minimum of 10 days before convergence and did not outperform the best maximum likelihood-based approach.
3 Extractive Summarization as a Contextual Bandit
BanditSum formulates extractive summarization as a contextual bandit, with documents as contexts and ordered sentence subsets as actions whose rewards measure agreement with gold summaries. Policy-gradient reinforcement learning trains a neural policy to sample summaries, using exploration and a self-critical baseline to optimize ROUGE-based rewards.
- Contextual-bandit formulation: Each document is a context, and each ordered subset of unique sentences is an action producing an extractive summary.The action contains M sentence indices, and its scalar reward measures the match between the induced extractive summary and the paired gold abstractive summary.
- Policy-gradient learning: The neural policy assigns probabilities to sentence-index sequences and is trained to favor sequences yielding high-quality summaries.The policy is parameterized by θ and defines a distribution over index sequences for each document.
- Policy-gradient learning: Policy-gradient learning estimates the objective gradient with sampled document-summary pairs and sampled index sequences because the selected indices are discrete.The resulting algorithm is an instance of REINFORCE.
- Policy structure and exploration: Sampling-without-replacement repeatedly normalizes affinities over unselected sentences and samples M unique sentences, with ε-greedy exploration during training.The policy is decomposed into a parameterized affinity function and a sampling distribution, avoiding systematic preference for earlier sentences.
- Variance reduction: A self-critical baseline subtracts the reward of greedy decoding, increasing a sampled sequence’s probability only when it outperforms greedy decoding.The baseline is r = R(i_greedy, a), and appropriately chosen baselines can reduce estimator variance without bias.
- Reward function: The reward averages ROUGE-2F1 and ROUGE-LF1, balancing precision and recall while optimizing multiple ROUGE variants.The reward function is designed to quantify overlap with the reference summary.
4 Model
The model encodes each document with bidirectional recurrent networks and decodes sentence representations into inclusion affinities. Processing the document bidirectionally lets sentence representations depend on the whole document, supporting redundancy-aware selection.
- Encoder: A word-level BiRNN builds sentence features, while a sentence-level BiRNN represents each sentence in document context.Word-level hidden states are averaged before the sentence-level recurrent encoding.
- Decoder: A multilayer perceptron maps sentence representations to sentence affinities for selection.The decoder operates on the contextual sentence representations produced by the encoder.
- Encoder: Bidirectional encoding allows each sentence representation to incorporate all other sentences, enabling affinities to depend on one another.This dependency is used to address summary-quality issues such as redundancy.
5 Experiments
Experiments use CNN, Daily Mail, and combined CNN/Daily Mail corpora, evaluate extractive systems with ROUGE F1, and compare BanditSum against established extractive baselines. The implementation uses recurrent encoders, Adam optimization, sampling-based gradient estimates, and three selected sentences at test time.
- Experimental setup: Experiments use CNN, Daily Mail, and combined CNN/Daily Mail datasets with standard training, validation, and test splits.Daily Mail contains 196,557 training, 12,147 validation, and 10,397 test documents; CNN contains 90,266, 1,220, and 1,093.
- Evaluation: Performance is evaluated using ROUGE-1, ROUGE-2, and ROUGE-L F1 scores against reference summaries.These metrics measure unigram, bigram, and longest-common-subsequence overlap, respectively.
- Baselines: BanditSum is compared with Lead-3, SummaRuNNer, Refresh, RNES, DQN, and NN-SE extractive systems.Lead-3 produces the document’s first three sentences as its summary.
- Model settings: The neural implementation uses one-layer word-level and two-layer sentence-level BiLSTMs, with 200 hidden units per direction.It initializes embeddings with 100-dimensional GloVe vectors and uses a 100-dimensional decoder hidden layer.
- Optimization: Training uses Adam with learning rate 5e-5, gradient clipping at 1, and B = 20 samples per update.Reported system performance is obtained within two epochs of training.
- Decoding: At test time, sentences are sorted by predicted probability until the length limit, with M = 3 sentences selected per summary.The value M = 3 follows validation results and prior experimental settings.
6 Experiment Results
The experiments evaluate BANDITSUM against extractive baselines using ROUGE, human judgments, and learning curves. Results cover benchmark performance, summary quality, convergence, and training efficiency.
- ROUGE Evaluation: BANDITSUM is compared with extractive baselines on CNN/Daily Mail using ROUGE evaluation and validation learning curves.The experiments also include qualitative human evaluation and compare BANDITSUM’s validation curve with SummaRuNNer.
- ROUGE Evaluation: BANDITSUM performs significantly better than Refresh, DQN, and SummaRuNNer, while slightly outperforming RNES on ROUGE-1 and slightly underperforming it on ROUGE-2.Unlike RNES, BANDITSUM does not require maximum-likelihood pre-training on heuristically generated extractive labels.
- ROUGE Evaluation: Policy-gradient baseline choices affect learning speed but do not significantly affect asymptotic performance.Average reward produced the fastest learning, while omitting a baseline reduced performance by about 2 points of ROUGE on average.
- Human Evaluation: Human judges rate BANDITSUM higher than Refresh and SummaRuNNer in overall quality and non-redundancy.The evaluations use pairwise preferences and rank summaries across overall quality, coverage, and non-redundancy.
- Learning Curve: BANDITSUM converges significantly faster than SummaRuNNer and exhibits less performance variance during training.Both models remain far from the estimated upper bound, indicating room for improvement.
- Training Efficiency: 25.5 hours per epoch and about 76 hours for three epochs are reported for BANDITSUM on a TITAN Xp.DQN took about 10 days to train on a GTX 1080, while Refresh took about 12 hours on one GPU, excluding its ROUGE pre-computation time.
7 Discussion: Contextual Bandit Setting Vs. Sequential Full RL Labeling
BANDITSUM uses contextual-bandit sampling to avoid the order bias of sequential binary labeling, especially when good summary sentences appear late. Experiments on early- and late-sentence datasets show faster convergence and a significant advantage on Dlate.
- Contextual-bandit modeling avoids sequential decision-order bias and is intended to improve selection when good sentences appear late.BANDITSUM samples sentences without replacement using affinity scores, while sequential labeling conditions later decisions on earlier ones.
- BANDITSUM, RNES, and RNES3 were trained without extractive labels on datasets where good sentences appeared early or late.The datasets contained 50 early-summary articles and 50 late-summary articles selected from 1,000 ordered documents.
- BANDITSUM has a significant advantage over RNES and RNES3 on Dlate, where good summary sentences appear late.Figure 2 compares average ROUGE-1,2,L F1 scores across ten trials.
- BANDITSUM finds a better solution more quickly than RNES and RNES3 on both early- and late-summary datasets.Each model used random initialization, ten training runs, and 100 epochs.
8 Conclusion
The paper presents BANDITSUM as a contextual-bandit framework for neural extractive summarization trained with reinforcement learning. It avoids sentence-level extractive labels, optimizes ROUGE against abstractive references, performs competitively, and converges with fewer update steps.
- BANDITSUM is a neural contextual-bandit framework for extractive summarization trained with reinforcement learning.
- BANDITSUM does not require sentence-level extractive labels and optimizes ROUGE against abstractive reference summaries.
- BANDITSUM performs better than or comparably to state-of-the-art extractive models that require extractive-label pre-training.
- BANDITSUM converges using significantly fewer update steps than competing approaches.
A Samples
The examples compare summaries generated by BANDITSUM and Refresh, illustrating differences in redundancy and source-information selection.
- Refresh selected a first sentence containing redundant detention-facility information, whereas BANDITSUM selected a sentence with additional source information.
- BANDITSUM tends to select more precise information than Refresh in the example summaries.