PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Encoding for Abstractive Sentence Summarization

Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou

arXiv:1704.07073v1cs.CL

TL;DR

Abstractive sentence summarization must retain salient information while removing unnecessary content, unlike machine translation’s preservation of input information. SEASS adds an explicit selection phase through a selective gate between recurrent encoding and attention-based decoding, and it outperforms state-of-the-art baselines across three test sets.

  • Problem

    Abstractive sentence summarization must keep highlights and remove unnecessary information, without the explicit alignment available in machine translation.

  • Method

    SEASS uses recurrent sentence encoding, a selective gate that constructs a tailored second-level representation, and attention-equipped decoding.

  • Results

    SEASS outperforms state-of-the-art baselines on English Gigaword, DUC 2004, and MSR-ATC test sets.

  • Takeaways & Limitations

    Selective encoding benefits abstractive sentence summarization across sentence lengths and determines important words before decoding.

Abstract

from arXiv · show

We propose a selective encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization. It consists of a sentence encoder, a selective gate network, and an attention equipped decoder. The sentence encoder and decoder are built with recurrent neural networks. The selective gate network constructs a second level sentence representation by controlling the information flow from encoder to decoder. The second level representation is tailored for sentence summarization task, which leads to better performance. We evaluate our model on the English Gigaword, DUC 2004 and MSR abstractive sentence summarization datasets. The experimental results show that the proposed selective encoding model outperforms the state-of-the-art baseline models.

1 Introduction

Abstractive sentence summarization must select salient information while filtering secondary content before generating a fluent summary. SEASS explicitly models this selection within a three-phase encoding, selection, and decoding framework.

  • Prior neural approaches: Earlier neural systems applied CNN or RNN encoders and neural decoders within an encoding-decoding paradigm.The cited work progresses from CNN-based models to recurrent sequence-to-sequence systems.
  • Motivation: Attention-based models dynamically extract encoded token information and align it with output tokens.This approach has been broadly successful in machine translation, but summarization requires information selection rather than preserving all input information.
  • Motivation: Abstractive sentence summarization requires selecting highlights while filtering out secondary information before decoding the summary.The proposed workflow explicitly separates encoding, selection, and decoding.
  • Proposed approach: SEASS combines an RNN sentence encoder, selective gate network, and attention-equipped decoder in three phases.The gate constructs a second-level representation and controls information flow from encoder to decoder.
  • Results: 17.54 ROUGE-2 F1, 9.56 ROUGE-2 recall, and 10.63 ROUGE-2 F1 are reported on English Gigaword, DUC 2004, and MSR-ATC, respectively.The model is evaluated on all three datasets and is reported to improve over state-of-the-art methods.

2 Related Work

Related work includes rule-based, statistical, neural, copying, segmentation, and metric-optimization approaches to sentence summarization. Neural sequence-to-sequence models established strong baselines, while later work explored additional mechanisms and training objectives.

  • Earlier approaches: Sentence summarization has been addressed with linguistic rules and statistical machine translation techniques.Statistical headline generation was trained using 8000 article-headline pairs.
  • Neural summarization: Rush et al. constructed large-scale English Gigaword parallel data and proposed ABS with an attentive CNN encoder and neural language-model decoder.ABS produced state-of-the-art results on the Gigaword and DUC 2004 test sets.
  • Neural summarization: Chopra et al. extended ABS by replacing its decoder with recurrent neural networks.This retained the CNN encoder while changing the decoder architecture.
  • Copying mechanisms: CopyNet, switch gates, and related copy mechanisms control whether outputs are copied from the input or generated from the decoder vocabulary.These methods adapt copying or gating ideas to response generation and summarization.
  • Alternative modeling and training: Segment-to-segment transduction introduced latent segmentation for input-output token correspondences and achieved performance comparable to ABS.Minimum Risk Training was also applied to optimize summarization evaluation metrics directly.

3 Problem Formulation

The task maps an input sentence to a shorter output summary. It distinguishes extractive summaries, whose words come from the input, from abstractive summaries, which may introduce words not present in the input.

  • Task definition: Given input x = (x_1, x_2, ..., x_n), the system produces summary y = (y_1, y_2, ..., y_l), where l ≤ n.Input tokens belong to the source vocabulary V_s, while summary tokens belong to the target vocabulary V_t.
  • Extractive summarization: Extractive sentence summarization requires every summary word to appear in the input sentence.This is expressed as |y| ⊆ |x|.
  • Abstractive summarization: Abstractive sentence summarization allows summary words that do not all appear in the input sentence.This is expressed as |y| ⊈ |x|, and the paper focuses on this task.

4 Model

SEASS extends sequence-to-sequence abstractive sentence summarization with a selective mechanism between recurrent encoding and attention-based decoding. Its gate filters encoder representations using sentence-level meaning to provide the decoder with tailored information.

  • 4 Model: SEASS uses a three-stage pipeline: a BiGRU sentence encoder, a selective gate network, and an attention-equipped GRU decoder.The encoder builds word representations, the gate selects encoded information, and the decoder generates the summary.
  • 4.1 Sentence Encoder: The BiGRU reads sentence embeddings in both directions and concatenates the resulting hidden states into basic sentence representations.Forward and backward GRUs process the input left-to-right and right-to-left, respectively.
  • 4.2 Selective Mechanism: The selective gate combines each BiGRU word representation with a sentence representation to construct a tailored representation for summarization.The gate generates a vector for each word and controls information flow from encoder to decoder.
  • 4.3 Summary Decoder: The decoder applies attention by matching its current state with each tailored encoder representation, normalizing importance scores, and computing a weighted context vector.The context vector is then combined with decoder inputs and state before predicting the next word.
  • 4.4 Objective Function: The model is trained to maximize output-summary probability through a negative log-likelihood objective over parallel sentence-summary pairs.Stochastic gradient descent with mini-batches learns the model parameters.

5 Experiments

The experiments evaluate SEASS on three abstractive sentence-summarization test sets using ROUGE variants and compare it with established baselines. SEASS outperforms the baselines across the reported datasets, with statistically significant gains reported on English Gigaword and DUC 2004.

  • Datasets: SEASS is evaluated on English Gigaword, DUC 2004, and MSR-ATC test sets.DUC 2004 and MSR-ATC are used only for testing because their training sets are too small for neural model training.
  • Evaluation Metric: ROUGE-1, ROUGE-2, and ROUGE-L are used to evaluate summarization quality, with F1 for English Gigaword and MSR-ATC and recall for DUC 2004.DUC 2004 uses ROUGE recall because reference summaries are capped at 75 bytes.
  • English Gigaword Results: 17.54 ROUGE-2 F1 is achieved by SEASS on the English Gigaword test set, a 6.22-point relative gain over ABS and a 1.57-point improvement over CAs2s.The reported SEASS result with beam search significantly outperforms the compared systems according to the official ROUGE script.
  • DUC 2004 Results: 29.21, 9.56, and 25.51 are achieved for ROUGE-1, ROUGE-2, and ROUGE-L recall on DUC 2004, respectively.SEASS significantly exceeds ABS+ by 1.07 ROUGE-2 recall and uses only English Gigaword sentence-summary training data.
  • MSR-ATC Results: 10.63 ROUGE-2 F1 is achieved on MSR-ATC, exceeding the s2s+att baseline by 1.02 points.The paper reports these results as the first ROUGE scores reported for MSR-ATC.

6 Discussion

The discussion evaluates selective encoding across sentence lengths and visualizes how the selective gate contributes to output summaries. SEASS consistently outperforms the s2s+att baseline, while the heat map shows that important words are selected before decoding.

  • Effectiveness of Selective Encoding: SEASS remains above s2s+att across the tested sentence-length groups, with especially large improvements for lengths 16, 20, 24, 32, 56, and 60.The English Gigaword test sentences range from 10 to 80 words; 18 groups were formed, with the first 14 plotted.
  • Effectiveness of Selective Encoding: Selective encoding benefits abstractive sentence summarization across input-length groups.The authors interpret the improvements observed across all groups as evidence that selective encoding benefits the task.
  • Effectiveness of Selective Encoding: Figure 4 compares ROUGE-2 F1 across input-sentence-length groups for s2s+att and SEASS on English Gigaword test sets.The figure organizes results by sentence-length group.
  • Saliency Heat Map of Selective Gate: The selective-gate analysis approximates each gate’s contribution to a summary output using the first derivative of the output score.Because the gate output is high-dimensional, the analysis visualizes contribution rather than all gate values directly.
  • Saliency Heat Map of Selective Gate: The heat map identifies important input words such as “europe,” “slammed,” “unacceptable,” “conditions,” and “france” as selected by the gate.The visualization indicates that the gate determines word importance before the decoder produces the summary.

7 Conclusion

The paper proposes selective encoding for abstractive sentence summarization, adding selection between encoding and decoding. Experiments on English Gigaword, DUC 2004, and MSR-ATC report improved performance over state-of-the-art methods.

  • 7 Conclusion: The selective encoding model extends sequence-to-sequence summarization with encoding, selection, and decoding phases.Its selective mechanism is described as mimicking human summarizers’ selection of important information before writing a summary.
  • 7 Conclusion: The end-to-end neural model uses a selective mechanism to choose important information before summary generation.This mechanism is integrated into the proposed selective encoding architecture.
  • 7 Conclusion: The model greatly improves performance over state-of-the-art methods on English Gigaword, DUC 2004, and MSR-ATC test sets.The conclusion reports this outcome across all three evaluated test sets.
Loading 1704.07073v1…