PaperlayerView exact revision on arXiv

Source-linked AI summary

Get To The Point: Summarization with Pointer-Generator Networks

Abigail See, Peter J. Liu, Christopher D. Manning

arXiv:1704.04368v2cs.CL

TL;DR

Abstractive summarization models can reproduce facts inaccurately and repeat themselves. This paper combines pointer-generator copying with coverage to address these issues, outperforming the abstractive state of the art by at least 2 ROUGE points on CNN/Daily Mail.

  • Problem

    Abstractive summarization systems can inaccurately reproduce factual details, mishandle out-of-vocabulary words, and repeat themselves, especially in multi-sentence summaries.

  • Method

    The model combines a hybrid pointer-generator network that copies source words or generates vocabulary words with coverage that tracks prior attention to reduce repetition.

  • Results

    At least 2 ROUGE points higher than the abstractive state of the art, the model performs better on the CNN/Daily Mail multi-sentence summarization task.

  • Takeaways & Limitations

    The architecture reduces inaccuracies and repetition while retaining abstractive abilities, although achieving higher levels of abstraction remains open.

  • Takeaways & Limitations

    The model is not close to producing highly abstractive summaries, indicating that its abstraction remains limited.

Abstract

from arXiv · show

Neural sequence-to-sequence models have provided a viable new approach for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text). However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves. In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways. First, we use a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator. Second, we use coverage to keep track of what has been summarized, which discourages repetition. We apply our model to the CNN / Daily Mail summarization task, outperforming the current abstractive state-of-the-art by at least 2 ROUGE points.

1 Introduction

Abstractive summarization enables novel wording but can reproduce facts inaccurately, mishandle out-of-vocabulary words, and repeat itself. This paper addresses these issues for longer multi-sentence summaries with a hybrid pointer-generator network and coverage on the CNN/Daily Mail dataset.

  • Motivation: Extractive methods select source passages, whereas abstractive methods can generate novel words and phrases, enabling paraphrasing, generalization, and real-world knowledge.Extractive summaries preserve baseline grammaticality and accuracy, while abstractive frameworks support more sophisticated capabilities.
  • Motivation: Recent sequence-to-sequence systems made abstractive summarization viable but inaccurately reproduce facts, mishandle OOV words, and repeat themselves.These shortcomings are illustrated in Figure 1’s comparison of abstractive summarization models.
  • Task: The paper targets longer multi-sentence summaries, arguing they are more challenging and useful than headline-generation tasks, and evaluates its model on CNN/Daily Mail.Longer summaries require greater abstraction while avoiding repetition.
  • Contributions: The hybrid pointer-generator copies source words via pointing to improve accuracy and OOV handling while retaining the ability to generate new words.The architecture balances extractive and abstractive approaches and had previously related applications in short-text summarization.
  • Contributions: The proposed coverage vector tracks and controls source-document coverage, and the authors show it is remarkably effective at eliminating repetition.Coverage is a novel variant adapted from neural machine translation.

2 Our Models

The models combine a baseline attentional sequence-to-sequence system with a pointer-generator that can copy source words, while coverage tracks prior attention to discourage repetition. The pointer-generator also permits out-of-vocabulary source words that the baseline’s fixed vocabulary cannot produce.

  • Baseline model: The baseline uses a bidirectional LSTM encoder, a unidirectional LSTM decoder, attention over encoder states, and a vocabulary distribution for prediction.The decoder uses the previous reference-summary word during training and the previous generated word at test time.
  • Pointer-generator model: The pointer-generator is a hybrid model that uses a soft generation probability to choose between generating vocabulary words and copying source words through attention.Its final distribution is defined over the union of the fixed vocabulary and all words appearing in the source document.
  • Pointer-generator model: The pointer-generator can produce out-of-vocabulary words, whereas the baseline is restricted to its preset vocabulary.For an out-of-vocabulary word, the vocabulary probability is zero, while the pointer mechanism can assign probability through the source attention distribution.
  • Coverage mechanism: Coverage maintains a vector equal to the sum of previous attention distributions, representing how much attention each source word has received.The initial coverage vector is zero because no source content has been covered at the first timestep.
  • Coverage mechanism: Coverage feeds prior attention information into the current attention mechanism and adds an overlap penalty to discourage repeatedly attending to the same source locations.The coverage loss is reweighted by λ and added to the primary loss; unlike machine translation, summarization does not require uniform coverage.

3 Related Work

Prior work developed neural abstractive summarization, pointer-based copying, and coverage mechanisms, while exposing limitations in long-text datasets and repetition control. This work builds on these approaches with a hybrid pointer-generator and a simpler attention-summing coverage method.

  • Neural abstractive summarization: Rush et al. (2015) introduced modern neural abstractive summarization, and subsequent work augmented attention-based models with recurrent decoders, meaning representations, hierarchical networks, and variational autoencoders.
  • Neural abstractive summarization: Large-scale long-text summarization datasets are rare; Nallapati et al. (2016) adapted the DeepMind question-answering dataset into CNN/Daily Mail and established the first abstractive baselines.
  • Pointer-generator networks: Pointer networks generate sequences from source elements using attention, and hybrid pointer approaches have been applied to machine translation, language modeling, and summarization.
  • Pointer-generator networks: The proposed model differs from earlier pointer methods by explicitly computing pgen, reusing attention as the copy distribution, and summing probabilities for repeated source words.
  • Coverage: Coverage methods track prior attention, and this work finds that simply summing attention distributions suffices, unlike GRU-based coverage updates used in earlier neural machine translation models.
  • Coverage: Temporal attention was tried for translation and summarization but was found too destructive because it distorted attention signals and reduced performance.

4 Dataset

The study uses the CNN/Daily Mail dataset of online news articles paired with multi-sentence summaries. Its version contains 287,226 training pairs, 13,368 validation pairs, and 11,490 test pairs.

  • Dataset: The CNN/Daily Mail dataset pairs online news articles with multi-sentence summaries.Articles average 781 tokens, while summaries average 3.75 sentences or 56 tokens.
  • Dataset: The dataset version contains 287,226 training pairs, 13,368 validation pairs, and 11,490 test pairs.The authors used scripts supplied by Nallapati et al. (2016) to obtain this version.
  • Dataset: Published CNN/Daily Mail results use the dataset’s anonymized version.

5 Experiments

Experiments use compact model dimensions and controlled sequence lengths, with pointer-generator and coverage variants evaluated under distinct training and decoding settings. Training comparisons show faster optimization for pointer-generator models, while coverage is most effective when introduced in a separate phase.

  • All models use 256-dimensional hidden states and 128-dimensional word embeddings; pointer-generator models use 50k-word source and target vocabularies, while baselines also use 150k vocabularies.
  • Articles are truncated to 400 tokens, summaries to 100 tokens during training and 120 tokens at test time, and decoding uses beam search with beam size 4.The 120-token test limit is almost never reached because beam search is self-stopping.
  • Pointer-generator training takes less than 230,000 iterations and 3 days 4 hours, compared with about 600,000 iterations for baseline models.Baseline training takes 4 days 14 hours with a 50k vocabulary and 8 days 21 hours with a 150k vocabulary.
  • Training coverage without its loss function produces no discernible reduction in repetition, while introducing coverage from the first iteration reduces overall performance.The early coverage objective interferes with the main objective, so coverage is trained as a separate phase.

6 Results

The pointer-generator model substantially improves over baselines, while adding coverage further improves scores and nearly eliminates repetition. The best model surpasses the prior abstractive system but not the lead-3 baseline or best extractive system, and ROUGE comparisons are limited by dataset differences.

  • Evaluation caveat: ROUGE scores are not strictly comparable because the models use plain-text summaries whereas Nallapati et al. (2016; 2017) use anonymized summaries.The original-text dataset may generally yield higher ROUGE scores than the anonymized dataset.
  • Results: Baseline models perform poorly on ROUGE and METEOR, and increasing vocabulary size from 50k to 150k does not help.Baseline summaries also frequently reproduce factual details incorrectly.
  • Results: The pointer-generator model achieves much better ROUGE and METEOR scores than the baseline while usually copying factual details correctly and handling out-of-vocabulary words.Repetition remains common in this model.
  • Results: Coverage further improves ROUGE and METEOR scores, convincingly surpassing Nallapati et al. (2016), the best prior abstractive model.The coverage model also nearly eliminates the pointer-generator model’s repetition problem.
  • Results: about 1% of total training time is devoted to coverage training, yet repetition is almost completely eliminated.This reduction is supported qualitatively and quantitatively.
  • Results: The best model does not surpass the lead-3 baseline or Nallapati et al. (2017), the current best extractive model.The paper defers discussion of this issue to Section 7.1.

7 Discussion

The discussion argues that extractive systems and the lead-3 baseline are difficult to beat because news structure and ROUGE favor early, source-faithful content. Although the pointer-generator produces some abstraction while improving factual copying, it remains heavily extractive and rarely generates highly abstractive summaries.

  • ROUGE and extractive strength: Extractive systems generally achieve higher ROUGE than abstractive systems, while even the best extractive system only slightly beats the strong lead-3 baseline.The authors attribute this partly to news articles placing important information near the beginning.
  • ROUGE and extractive strength: 400 tokens, or about 20 sentences, yield significantly higher ROUGE scores than 800 tokens, supporting the strength of early-content selection.The lead-3 baseline is also favored because reference summaries are subjective and ROUGE rewards matching one reference summary’s wording and content.
  • ROUGE and extractive strength: All models gain over 1 METEOR point from stem, synonym, and paraphrase matching, suggesting some abstraction, but none surpasses the lead-3 baseline.ROUGE’s single-reference setup can score valid abstractive alternatives poorly, such as assigning 0 ROUGE to an acceptable paraphrase.
  • Model abstractiveness: 35% of the final model’s summaries copy whole article sentences, compared with 1.3% for reference summaries, although the remaining 65% uses truncation and recombination.The pointer mechanism makes factual copying more reliable, but sentence extraction remains a major area for improvement.
  • Model abstractiveness: pgen rises from about 0.30 during training to about 0.53, then averages 0.17 at test time, indicating substantial test-time preference for copying.The model can produce novel n-grams and more impressive abstractions, especially in sports-story templates, but does not routinely generate highly abstractive summaries.

8 Conclusion

The paper presents a hybrid pointer-generator architecture with coverage that reduces inaccuracies and repetition, significantly outperforming the abstractive state of the art on a challenging long-text dataset. Despite exhibiting many abstractive abilities, achieving higher levels of abstraction remains open.

  • 8 Conclusion: The hybrid pointer-generator architecture with coverage reduces inaccuracies and repetition.The architecture combines pointer-generator mechanisms with coverage.
  • 8 Conclusion: The model significantly outperforms the abstractive state-of-the-art result on a new, challenging long-text dataset.The paper applies the model to a new long-text summarization dataset.
  • 8 Conclusion: The model exhibits many abstractive abilities, but attaining higher levels of abstraction remains an open research question.The conclusion identifies increased abstraction as unresolved future work.

Supplementary Material

The supplementary examples compare reference summaries with model outputs, illustrating how pointer-generator and coverage mechanisms improve factual accuracy and reduce repetition while retaining abstractive generation. They also show remaining cases where individually accurate sentences may not form a coherent whole.

  • Model comparisons: The examples show pointer-network models accurately reproducing out-of-vocabulary names and the final model generating the novel word defeated to combine source fragments.In the example, the baseline reports the wrong score and substitutes bedene for thiem, while pointer-network models reproduce thiem and aljaz accurately.
  • Model comparisons: The pointer-generator model fixes factual substitutions and fabricated details, while adding coverage fixes repetition.The baseline changes new zealand to dutch, fabricates irish, mishandles saili and auckland-based, and repeats content; pointer-generation improves accuracy and coverage addresses repetition.
  • Model comparisons: The final model can produce novel abstractive words, including beat and says, while omitting some source phrases.Both the baseline and final model use beat in a completely abstractive first sentence, and the final model uses says to paraphrase told cnn ’s christiane amanpour.
  • Model comparisons: The final model can produce individually accurate sentences that do not make sense as a whole.In one example, the baseline contains multiple factual inaccuracies, whereas the final model’s sentences are individually accurate but globally incoherent.
  • Model comparisons: The pointer-generator model more accurately reports a military reshuffle instead of fabricating an unmentioned U.N. peacekeeping detail.The baseline’s false detail likely reflects a training-data association between U.N. peacekeeping forces and northern sinai.
Loading 1704.04368v2…