PaperlayerView exact revision on arXiv

Source-linked AI summary

A Deep Reinforced Model for Abstractive Summarization

Romain Paulus, Caiming Xiong, Richard Socher

arXiv:1705.04304v3cs.CL

TL;DR

Long-document abstractive summarization suffers from repetitive, incoherent outputs and exposure bias in standard supervised training. The paper introduces intra-attention over input and generated output together with mixed supervised and reinforcement learning. It reports 41.16 ROUGE-1 on CNN/Daily Mail, higher readability in human evaluation, and state-of-the-art CNN/Daily Mail results, while noting that ROUGE alone may not guarantee readability or relevance.

  • Problem

    Attentional encoder-decoder summarizers perform well on short sequences but often produce repetitive phrases on longer documents and summaries.

  • Method

    The model combines encoder and decoder intra-attention with a mixed objective joining maximum-likelihood cross-entropy and policy-gradient reinforcement learning.

  • Results

    41.16 ROUGE-1 was achieved on CNN/Daily Mail, with higher readability and relevance scores for RL+ML summaries among the evaluated models.

  • Takeaways & Limitations

    The procedure improves long-sequence abstractive summarization readability and achieves state-of-the-art results on CNN/Daily Mail, while also being applied end-to-end to NYT.

  • Takeaways & Limitations

    Optimizing ROUGE does not guarantee improved output quality or readability because discrete metrics can be increased without actual gains in readability or relevance.

Abstract

from arXiv · show

Attentional, RNN-based encoder-decoder models for abstractive summarization have achieved good performance on short input and output sequences. For longer documents and summaries however these models often include repetitive and incoherent phrases. We introduce a neural network model with a novel intra-attention that attends over the input and continuously generated output separately, and a new training method that combines standard supervised word prediction and reinforcement learning (RL). Models trained only with supervised learning often exhibit "exposure bias" - they assume ground truth is provided at each step during training. However, when standard word prediction is combined with the global sequence prediction training of RL the resulting summaries become more readable. We evaluate this model on the CNN/Daily Mail and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the CNN/Daily Mail dataset, an improvement over previous state-of-the-art models. Human evaluation also shows that our model produces higher quality summaries.

1 INTRODUCTION

Abstractive summarization can generate new phrases, but earlier attentional encoder-decoder systems mainly handled short sequences and often repeated phrases on longer documents. The paper introduces intra-attention and mixed supervised–reinforcement learning to improve long-sequence summaries.

  • Abstractive systems generate new phrases, unlike extractive systems that copy parts of the input.
  • Earlier attentional encoder-decoder models mainly summarized one- or two-sentence inputs into even shorter summaries.
  • Longer CNN/Daily Mail inputs exposed a problem: generated summaries often contained unnatural repeated phrases.
  • The model combines encoder intra-temporal attention, decoder sequential intra-attention, and supervised cross-entropy with policy-gradient reinforcement learning.
  • 41.16 ROUGE-1 was achieved on CNN/Daily Mail, while human evaluation found the generated summaries more readable than other abstractive approaches.

2 NEURAL INTRA-ATTENTION MODEL

The model uses encoder and decoder intra-attention to track previously attended or generated content, while a pointer mechanism combines copying with token generation. Additional training and decoding choices target repetition and exposure bias.

  • A bidirectional LSTM encoder represents input tokens, and an LSTM decoder represents output tokens using shared embeddings.
  • 2.1 INTRA-TEMPORAL ATTENTION ON INPUT SEQUENCE: Intra-temporal attention penalizes input tokens that received high attention earlier, encouraging different input regions across decoding steps.
  • 2.2 INTRA-DECODER ATTENTION: Intra-decoder attention incorporates previously decoded states so long outputs can make structured predictions and avoid repeating distant information.
  • 2.3 TOKEN GENERATION AND POINTER: At each step, a switch chooses between generating a token with a softmax layer and copying an input token through the pointer mechanism.
  • 2.3 TOKEN GENERATION AND POINTER: The pointer mechanism uses temporal attention weights as the probability distribution for copying input tokens.
  • 2.3 TOKEN GENERATION AND POINTER: During testing, beam search assigns zero probability to outputs that would create a repeated trigram.

3 HYBRID LEARNING OBJECTIVE

The paper combines supervised maximum-likelihood training with reinforcement learning to address exposure bias and optimize sequence-level rewards. Its mixed objective is intended to improve the naturalness and readability of long generated summaries while retaining metric optimization.

  • Supervised learning: Teacher forcing trains each decoder step with the ground-truth prefix, creating exposure bias because that supervision is unavailable during testing.
  • Reinforcement learning: Reinforcement learning optimizes a discrete reward directly by comparing a sampled output with a greedy baseline output.The self-critical policy gradient algorithm is used for this comparison.
  • Caveat: Optimizing ROUGE alone may increase the metric without improving readability or relevance, motivating the inclusion of supervised language-model training.Human readability is described as better captured by language-model perplexity than by n-gram overlap alone.
  • Mixed objective: The mixed objective combines maximum-likelihood cross-entropy with policy-gradient reinforcement-learning rewards.The authors hypothesize that the conditional language-model component helps policy learning generate more natural summaries.
  • Mixed objective: The scaling factor γ accounts for the difference in magnitude between the reinforcement-learning and maximum-likelihood losses.

4 RELATED WORK

Related work spans extractive and abstractive summarization, neural encoder-decoder architectures, and reinforcement-learning methods for sequence generation. Prior evaluations largely focused on short summaries, motivating work on longer-document abstractive summarization.

  • Neural sequence models: Neural encoder-decoder models use recurrent networks to encode an input sequence and generate an output sequence with another recurrent network.These architectures have been applied to machine translation, summarization, and question answering.
  • Reinforcement learning: Reinforcement learning is applicable to sequence generation because evaluation metrics such as BLEU, ROUGE, and METEOR are not differentiable.
  • Reinforcement learning: REINFORCE-based sequence-generation training improved over supervised methods, while self-critical training removed the need for a separate critic model.
  • Summarization approaches: Extractive systems select and rearrange important input phrases, whereas abstractive systems can generate more novel sequences.
  • Prior summarization evaluations: Earlier abstractive summarization evaluations often used DUC-2004, whose systems typically handled one or two sentences and generated summaries up to 75 characters.CNN/Daily Mail work extended evaluation to longer inputs and summaries.

5 DATASETS

The experiments use modified CNN/Daily Mail data and NYT article-abstract pairs to evaluate abstractive summarization across complementary document and summary formats. Results are organized in quantitative tables and an example-output table.

  • CNN/Daily Mail: The modified CNN/Daily Mail dataset contains 287,113 training, 13,368 validation, and 11,490 testing examples.Inputs are limited to 800 tokens and outputs to 100 tokens, with average lengths of 632 and 53 tokens.
  • New York Times: The NYT dataset contains articles published between 1996 and 2007 and is evaluated through article-abstract pairs for end-to-end abstractive summarization.The paper describes this as the first such end-to-end abstractive evaluation on NYT.
  • Dataset comparison: CNN/Daily Mail and NYT provide complementary formats because NYT abstracts are shorter, more varied, and more paraphrastic than the article-like CNN/Daily Mail summaries.
  • Evaluation materials: Table 1 reports quantitative results for models on the CNN/Daily Mail test dataset.
  • Evaluation materials: Table 2 reports quantitative results for models on the NYT test dataset, while Table 3 presents processed outputs from three best models with an example-specific ROUGE score.

6 RESULTS

Experiments compare maximum-likelihood, reinforcement-learning, and mixed-objective training, alongside intra-decoder attention, across CNN/Daily Mail and NYT settings. The mixed objective improves both automatic scores and human-rated quality, while intra-attention helps mainly with longer summaries.

  • Quantitative results: Intra-decoder attention improves ROUGE scores on CNN/Daily Mail but not on NYT.
  • Quantitative analysis: Intra-attention increases ROUGE-1 for examples with long ground-truth summaries but decreases it for shorter summaries.
  • Quantitative results: Mixed-objective ML+RL training and RL achieve much higher scores than ML across the evaluated datasets.
  • Baseline comparisons: The mixed-objective model surpasses Nallapati et al. (2016), lead-3, and SummaRuNNer on CNN/Daily Mail.
  • Baseline comparisons: The mixed-objective model obtains higher ROUGE scores than See et al. (2017), although preprocessing differences make direct comparison difficult.
  • Baseline comparisons: On Durrett et al.'s NYT splits, the mixed-objective model outperforms their extractive model and extractive baselines.
  • Qualitative analysis: RL has the highest ROUGE scores but the lowest readability, whereas RL+ML has the highest human-rated readability and relevance.

7 CONCLUSION

The paper presents a model and training procedure that performs strongly on CNN/Daily Mail, improves readability, and suits long output sequences. It also applies abstractive summarization to NYT and cautions against optimizing ROUGE alone.

  • The model and training procedure achieve state-of-the-art text summarization results on CNN/Daily Mail.
  • The approach improves generated-summary readability and is better suited to long output sequences.
  • The study runs an abstractive summarization model on the NYT dataset for the first time.
  • ROUGE scores have shortcomings and should not be the only optimization metric for long-sequence summarization.

A.1 PREPROCESSING

NYT preprocessing constructs comparable article-summary examples by cleaning fields, splitting abstracts on semicolons, and limiting sequence lengths.

  • NYT examples concatenate headline, byline, and full article text into one input sequence after removing incomplete documents.
  • Abstract preprocessing removes selected markers and occurrences of photo, graph, chart, map, table, and drawing in specified positions.
  • Splitting NYT abstracts on semicolons creates multisentence summaries for a format and evaluation procedure similar to CNN/Daily Mail.
  • After preprocessing, examples average 549 input tokens and 40 output tokens, with input and output limits of 800 and 100 tokens.

A.2 DATASET SPLITS

The dataset uses chronological document splits for reproducibility and production-oriented evaluation, while NER and pointer supervision support entity and out-of-vocabulary handling during training.

  • Documents are sorted chronologically into 90% training, 5% validation, and 5% testing splits.The split sizes are 589,284 training, 32,736 validation, and 32,739 testing examples.
  • Chronological splitting is intended to make the dataset splits reproducible and reflect deployment on recent articles.
  • Named entity recognition identifies PERSON, LOCATION, ORGANIZATION, and MISC entities in abstracts and finds their first input occurrences for supervision.
  • NER is used only during dataset creation, while pointer supervision is added for out-of-vocabulary output tokens appearing in the input.

B HYPERPARAMETERS AND IMPLEMENTATION DETAILS

The implementation uses stochastic teacher forcing to reduce exposure bias, alongside specified model dimensions, vocabulary limits, embeddings, and optimization settings.

  • At each decoding step, the model uses the previously generated token instead of the ground-truth token with 25% probability.This modified teacher forcing procedure is intended to reduce exposure bias.
  • The ML+RL loss uses γ = 0.9984.
  • The architecture comprises two 200-dimensional bidirectional-encoder LSTMs and one 400-dimensional decoder LSTM.
  • Input and output vocabularies are capped at 150,000 and 50,000 tokens, respectively, using the most frequent training-set tokens.
  • Input embeddings are 100-dimensional and initialized with GloVe, while models are trained with Adam using batch size 50.The stated ML-training learning rate is α = 0.001.
Loading 1705.04304v3…