PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss

Wan-Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, Min Sun

arXiv:1805.06266v2cs.CL

TL;DR

Extractive summarization can select important content but may be less readable, while abstractive summarization can be readable yet lose or mistake facts. The paper unifies both through cross-level attention modulation and inconsistency loss, achieving state-of-the-art ROUGE scores and the strongest reported human-evaluation informativity and readability on CNN/Daily Mail.

  • Problem

    Extractive summaries can be less readable, whereas abstractive summaries may lose or mistake factual details.

    Source: §1 · p. 1

  • Method

    The model combines sentence-level and word-level attention, modulates word attention using sentence attention, and trains with inconsistency loss.

    Source: §1 · p. 2

  • Results

    The model achieves state-of-the-art ROUGE scores and is reported as the most informative and readable summarization on CNN/Daily Mail in human evaluation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The inconsistency loss enables extractive and abstractive summarization to be mutually beneficial.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

We propose a unified model combining the strength of extractive and abstractive summarization. On the one hand, a simple extractive model can obtain sentence-level attention with high ROUGE scores but less readable. On the other hand, a more complicated abstractive model can obtain word-level dynamic attention to generate a more readable paragraph. In our model, sentence-level attention is used to modulate the word-level attention such that words in less attended sentences are less likely to be generated. Moreover, a novel inconsistency loss function is introduced to penalize the inconsistency between two levels of attentions. By end-to-end training our model with the inconsistency loss and original losses of extractive and abstractive models, we achieve state-of-the-art ROUGE scores while being the most informative and readable summarization on the CNN/Daily Mail dataset in a solid human evaluation.

1 Introduction

Text summarization condenses text while preserving important points, with extractive methods favoring simplicity and abstractive methods favoring readable, concise generation. The paper unifies both approaches by combining sentence- and word-level attention and adding inconsistency loss.

  • Text summarization condenses text while maintaining important points for applications including news digests, search results, and reports.Source: §1 · p. 1
  • Extractive methods select source sentences and are typically simpler, whereas abstractive methods can generate novel words and phrases.Source: §1 · p. 1
  • Abstractive summaries can be more coherent and concise than extractive summaries, but abstractive systems may inaccurately reproduce factual details.Source: §1 · pp. 1–2
  • The unified model uses sentence-level attention to modulate word-level attention, making words in less attended sentences less likely to be generated.Source: §1 · p. 2
  • The proposed inconsistency loss encourages consistency between attention levels without requiring additional human annotation.Source: §1 · p. 2
  • The model achieves the best ROUGE scores on CNN/Daily Mail and outperforms recent state-of-the-art methods in informativity and readability in human evaluation.Source: §1 · p. 2

2 Related Work

Related work covers neural extractive and abstractive summarization, including mechanisms for handling unknown words and repetition. The paper combines sentence-level attention from an extractive model with word-level attention from an abstractive model and adds inconsistency loss.

Extractive summarization

  • Prior work includes neural models that represent sentences and select them for extractive summarization.Source: §2 · p. 2

Abstractive summarization

  • Pointer-generator and related models address out-of-vocabulary words and repeated phrases in generated summaries.Source: §2 · p. 3

Unified model

3 Our Unified Model

The unified model combines sentence-level extraction with word-level abstractive generation, using sentence attention to modulate word attention and an inconsistency loss to align them during training.

3 Our Unified Model

  • The model combines sentence-level attention from an extractor with dynamic word-level attention from an abstracter.The extractor produces sentence probabilities, while the abstracter generates summary text word by word.Source: §3 · p. 3

3.1 Combining Attentions

  • Sentence-level attention modulates word-level attention so words in less attended sentences are less likely to be generated.The attentions are combined by scalar multiplication and renormalization; high word attention requires both levels to be high.Source: §3.1 · pp. 3–4

3.2 Inconsistency Loss

  • The inconsistency loss encourages sentence attention to be high when word attention is high and makes the two attention levels mutually beneficial.It uses the top K attended words and summary length T, while the original extractor, abstracter, and coverage losses prevent degenerate attention distributions.Source: §3.2 · p. 4

3.3 Extractor

  • The extractor uses a hierarchical bidirectional GRU and predicts sentence-level attention, with labels selected to favor highly informative sentences.Ground-truth labels are constructed using ROUGE-L recall against the reference abstractive summary, prioritizing sentences that increase collective informativity.Source: §3.3 · p. 4

3.4 Abstracter

  • The updated word attention affects decoding by producing a context vector that updates the final word distribution.The abstracter uses a pointer-generator network that can copy article words or generate words from a fixed vocabulary, including OOV words.Source: Figure 4 · p. 5, §3.4 · p. 5

3.5 Training Procedure

  • End-to-end training minimizes extractor, abstractive, coverage, and inconsistency losses together.The authors also pre-train both components; during end-to-end training, sentence attention is soft and the extractor loss may receive a larger weight.Source: §3.5 · pp. 5–6

4 Experiments

The experiments use the CNN/Daily Mail news dataset and describe the model’s training and implementation settings, including pre-training and two-stage training.

  • The CNN/Daily Mail dataset contains news stories paired with human-written multi-sentence summaries, with anonymized and non-anonymized versions.Source: §4.1 · p. 6
  • The extractor and abstracter use 128-dimensional word embeddings, vocabularies of 50k, hidden dimensions of 200 and 256, Adagrad, early stopping, and summaries limited to 120 tokens.Source: §4.2 · p. 6

Pre-training

  • During pre-training, the extractor runs for 27k iterations, while the abstracter uses ground-truth extracted sentences and trains with and without coverage mechanisms.Source: §4.2 · p. 6

Two-stages training

  • During two-stage training, the abstracter receives sentences whose pre-trained extractor probabilities βn exceed 0.5 before fine-tuning.Source: §4.2 · p. 6

5 Results

The experiments evaluate extracted sentences, generated summaries, attention consistency, and human judgments. End-to-end training with inconsistency loss achieves strong ROUGE results, reduces inconsistency, and improves reported human-evaluation quality.

Extracted sentences

  • The extractor is evaluated using full-length ROUGE recall between selected sentences and reference abstractive summaries, selecting sentences with β greater than 0.5.Source: §5.1 · p. 6

Abstractive summarization

  • The two-stage model outperforms the pointer-generator model on ROUGE-1 and ROUGE-2, while end-to-end training with inconsistency loss exceeds the lead-3 baseline.Source: Table 2 · p. 7

Extracted sentences

  • The ground-truth extracted sentences are an upper bound but cannot reach 100% recall because abstractive references may contain words absent from the article.Source: §5.1 · p. 7, Table 1 · p. 7
  • The end-to-end extractor performs best when trained with inconsistency loss.Source: §5.1 · p. 7

Abstractive summarization

  • The end-to-end model without inconsistency loss already improves ROUGE through cooperation between the extractor and abstracter, while inconsistency loss yields state-of-the-art ROUGE scores.Source: §5.2 · p. 7

Attention consistency

  • The inconsistency rate counts decoder steps whose maximally attended word belongs to a sentence with below-mean sentence attention.Source: §5.2 · p. 7
  • 20% to 4%: inconsistency loss significantly decreases the average inconsistency rate on the test set.Source: §5.2 · p. 7

Human evaluation

  • Human evaluation scores informativity, conciseness, and readability on a 1-to-5 scale, using a random-summary trap to reject unreliable evaluations.Source: §5.3 · p. 8

6 Conclusion

The paper concludes that its unified model combines extractive and abstractive summarization, with inconsistency loss aligning their attentions. End-to-end training achieves strong ROUGE performance and favorable human-evaluation results on CNN/Daily Mail.

  • The unified model combines the strengths of extractive and abstractive summarization through sentence-level and word-level attention.Source: §6 · p. 9
  • The inconsistency loss penalizes disagreement between the two attention levels and enables extractive and abstractive summarization to be mutually beneficial.Source: §6 · p. 9
  • End-to-end training achieves the best reported ROUGE-recall and ROUGE results while producing the most informative and readable summaries in the CNN/Daily Mail human evaluation.Source: §6 · p. 9

Related papers

Loading 1805.06266v2…