PaperlayerView exact revision on arXiv

Source-linked AI summary

Ranking Sentences for Extractive Summarization with Reinforcement Learning

Shashi Narayan, Shay B. Cohen, Mirella Lapata

arXiv:1802.08636v2cs.CL

TL;DR

Extractive summarization must select informative sentences while preserving document content, but sentence-level cross-entropy training does not directly match ranking or ROUGE evaluation. The paper uses reinforcement learning to optimize ROUGE globally, and reports that its model outperforms state-of-the-art extractive and abstractive systems on CNN and DailyMail in automatic and human evaluations.

  • Problem

    Cross-entropy training does not directly optimize sentence ranking for summary generation or the ROUGE metric, and sentence labels are extrapolated from abstractive summaries.

  • Method

    The paper combines maximum-likelihood cross-entropy with policy-gradient reinforcement learning to globally optimize ROUGE while ranking sentences.

  • Results

    The model outperforms state-of-the-art extractive and abstractive systems on CNN and DailyMail in automatic ROUGE and human evaluations.

  • Takeaways & Limitations

    Reinforcement learning steers extractive summarization toward informative, fluent, and concise summaries within the evaluated CNN and DailyMail setting.

  • Takeaways & Limitations

    The human evaluation excluded several systems whose outputs were not comparable because they used named-entity-anonymized CNN and DailyMail data.

Abstract

from arXiv · show

Single document summarization is the task of producing a shorter version of a document while preserving its principal information content. In this paper we conceptualize extractive summarization as a sentence ranking task and propose a novel training algorithm which globally optimizes the ROUGE evaluation metric through a reinforcement learning objective. We use our algorithm to train a neural summarization model on the CNN and DailyMail datasets and demonstrate experimentally that it outperforms state-of-the-art extractive and abstractive systems when evaluated automatically and by humans.

1 Introduction

The paper frames extractive summarization as sentence ranking and argues that cross-entropy training does not align with ranking and ROUGE-based evaluation. It proposes reinforcement learning to globally optimize ROUGE and reports stronger automatic and human evaluations.

  • Extractive systems concatenate the most important document sentences, but cross-entropy models do not necessarily learn the ranking needed for summaries.
  • Cross-entropy training is argued to produce verbose summaries with unnecessarily long sentences and redundant information.
  • The proposed model combines maximum-likelihood cross-entropy with policy-gradient rewards to directly optimize ROUGE during sentence ranking.
  • The model uses a hierarchical document encoder and hierarchical sentence extractor within its neural summarization architecture.
  • On CNN and DailyMail, the model outperforms state-of-the-art extractive and abstractive systems in ROUGE and human evaluations.
  • Human evaluations found the model’s summaries more informative and complete across preference and document-information assessments.

2 Summarization as Sentence Ranking

The model represents documents hierarchically, labels sentences for inclusion, ranks them by extraction confidence, and assembles summaries from selected sentences. Reinforcement learning directly optimizes ROUGE so sentences are favored when they occur in high-scoring summaries.

  • Summarization as Sentence Ranking: The summarizer selects m sentences from n document sentences, assigning each sentence a relevance probability for summary inclusion.
  • Sentence Encoder: A convolutional sentence encoder converts sentences into continuous representations, using multiple kernel sizes and max-pooling over feature maps.
  • Document Encoder: An LSTM document encoder composes the sentence sequence into a document representation and processes sentences in reverse order.
  • Sentence Extractor: An LSTM sentence extractor labels each sentence as relevant or irrelevant, conditioning predictions on the document and previously labeled sentences.
  • Reinforcement Learning: Figure 1 shows candidate summaries assembled from top-ranked sentences, with rewards comparing each candidate against the gold summary.
  • Reinforcement Learning: Reinforcement learning directly optimizes ROUGE and ranks sentences higher when they frequently occur in high-scoring summaries.

3 The Pitfalls of Cross-Entropy Loss

The paper identifies mismatches between cross-entropy training, sentence-level labels, summary ranking, and ROUGE evaluation. Candidate summaries can achieve similar high ROUGE scores while differing in concision and redundancy, motivating global optimization.

  • Training-Objective Discrepancies: Maximum-likelihood training optimizes ground-truth sentence-label likelihood, whereas summarization requires ranking sentences and is evaluated with ROUGE.
  • Label Discrepancies: Sentence-level labels are extrapolated from abstractive summaries, so they do not naturally identify which sentences should be extracted.
  • Label Discrepancies: Individually labeling sentences can create too many positive labels, while collective labels may cause cross-entropy models to underfit by ignoring other sentences.
  • Candidate Summaries: 57.5%, 57.2%, and 57.1% are the mean ROUGE-1, ROUGE-2, and ROUGE-L F1 scores for the second-, third-, and fourth-ranked candidate summaries.
  • Candidate Summaries: A sentence with a high individual ROUGE score can still be absent from high-scoring summaries because it overlaps with other content and increases redundancy.

4 Sentence Ranking with Reinforcement Learning

The paper adapts reinforcement learning to rank sentences for extractive summarization by optimizing ROUGE over sampled summaries rather than relying solely on ground-truth label likelihood. It restricts training to high-probability candidate extracts to make this optimization tractable and faster.

  • 4 Sentence Ranking with Reinforcement Learning: Reinforcement learning combines cross-entropy loss with policy-gradient rewards to globally optimize ROUGE for sentence ranking.The reward encourages sentences that frequently occur in high-scoring summaries.
  • 4 Sentence Ranking with Reinforcement Learning: The model treats documents as environments, sentence relevance scores as policy outputs, sampled ranked extracts as actions, and ROUGE agreement with gold summaries as reward.The reward is based on how well the extract resembles the gold-standard summary.
  • 4.1 Policy Learning: A single sampled extract approximates the expected policy gradient because enumerating all possible extracts is prohibitive.The approximation uses one sample from the model distribution for each training example in a batch.
  • 4.2 Training with High Probability Samples: The training procedure limits exploration to high-probability summaries and approximates this set using the k extracts with highest ROUGE scores.Candidate summaries are assembled from document sentences that individually receive high ROUGE scores.
  • 4.2 Training with High Probability Samples: Precomputed high-scoring candidates avoid an online reward estimator and substantially speed training relative to MIXER and related schemes.The authors also report faster convergence to an optimal policy from their gradient approximation.

5 Experimental Setup

The experiments evaluate REFRESH on CNN and DailyMail using automatic ROUGE scores and two human studies covering summary preference and retained key information. The setup compares REFRESH with leading-sentence, neural cross-entropy, and abstractive systems under dataset-specific extraction settings.

  • Summarization Datasets: REFRESH is evaluated on the CNN and DailyMail news highlights datasets using their standard training, validation, and test splits.The splits contain 90,266/1,220/1,093 CNN documents and 196,961/12,148/10,397 DailyMail documents.
  • Implementation Details: The system selects three sentences for CNN articles and four for DailyMail articles, reflecting the datasets’ average gold-highlight lengths.High-scoring extracts are estimated from ten document sentences with the highest ROUGE scores.
  • Evaluation: Automatic evaluation reports F1 ROUGE-1 and ROUGE-2 for informativeness and ROUGE-L for fluency against LEAD and neural cross-entropy systems.The comparison includes a baseline selecting the first m sentences and two similar neural models.
  • Evaluation: Human preference evaluation ranks LEAD, See et al. (2017), REFRESH, and human-authored highlights for informativeness and fluency.Five participants evaluated 20 articles: ten from CNN and ten from DailyMail.
  • Evaluation: A second human study measures retained key information by asking participants to answer fact-based questions derived from gold summaries.It uses 71 questions across the same 20 documents and averages scores for correct, partially correct, and incorrect answers.

6 Results

REFRESH outperforms extractive and abstractive systems on automatic ROUGE evaluation and is preferred over competing systems in human evaluations, while preserving more answerable information.

  • Table 2 reports ROUGE-1, ROUGE-2, and ROUGE-L F1 scores separately for extractive and abstractive systems on CNN and DailyMail test sets.
  • REFRESH is superior to the LEAD baseline and extractive systems across datasets and ROUGE metrics.
  • REFRESH also outperforms the evaluated abstractive systems, while See et al. (2017) copies more than 35% of source sentences.
  • Human-authored summaries rank first 39% of the time, while REFRESH ranks second best overall.
  • Participants answer 66.34% of questions correctly from REFRESH summaries, compared with 36.33% for LEAD and 28.73% for See et al. (2017).
  • QA evaluation amplifies system differences because it targets users’ information needs, whereas ROUGE measures surface n-gram similarity.

7 Related Work

Prior work ranks sentences using manually defined or independently learned salience features, while reinforcement learning has mainly addressed multi-document selection and text rewriting rather than extractive sentence ranking.

  • Traditional summarization methods manually define sentence-ranking features, and many later methods score sentences independently.
  • Reinforcement learning has been used for selecting sentences or subsets from multi-document clusters through search with delayed rewards.
  • REINFORCE has improved encoder-decoder text rewriting by directly optimizing non-differentiable objectives or injecting task-specific constraints.
  • The paper identifies no prior attempt to use reinforcement learning to train a sentence ranker for extractive summarization.

8 Conclusions

The paper globally trains an extractive model to optimize ROUGE, using reinforcement learning to explore candidate summaries and produce informative, fluent, concise outputs on CNN and DailyMail.

  • The model is globally trained by optimizing the ROUGE evaluation metric through reinforcement learning.
  • The training algorithm explores candidate summaries while learning to optimize a task-relevant reward function.
  • Experiments show the model outperforms state-of-the-art extractive and abstractive systems on the CNN and DailyMail datasets.
  • Future work will consider smaller discourse units and jointly model compression and extraction.
Loading 1802.08636v2…