Source-linked AI summary
Global Encoding for Abstractive Summarization
Junyang Lin, Xu Sun, Shuming Ma, Qi Su
TL;DR
Neural abstractive summarization models can generate repetitive and semantically irrelevant summaries. This paper introduces global encoding with a convolutional gated unit to refine source representations using global context, and reports stronger results on LCSTS and Gigaword with reduced repetition.
Problem
Conventional attention-based seq2seq models for abstractive summarization can suffer from repetition and semantic irrelevance.
Method
A convolutional gated unit performs global encoding over encoder outputs to refine source representations using global context.
Results
The model outperforms baselines on LCSTS and Gigaword, achieving ROUGE-2 F1 scores of 26.8 and 17.8, respectively.
Takeaways & Limitations
The analysis indicates that the model reduces repetition and generates summaries with higher diversity than conventional seq2seq.
Abstract
from arXiv · showhide
In neural abstractive summarization, the conventional sequence-to-sequence (seq2seq) model often suffers from repetition and semantic irrelevance. To tackle the problem, we propose a global encoding framework, which controls the information flow from the encoder to the decoder based on the global information of the source context. It consists of a convolutional gated unit to perform global encoding to improve the representations of the source-side information. Evaluations on the LCSTS and the English Gigaword both demonstrate that our model outperforms the baseline models, and the analysis shows that our model is capable of reducing repetition.
1 Introduction
Attention-based seq2seq summarization can produce repetition and semantically irrelevant summaries because encoder outputs may contain noise for attention. The proposed global encoding model refines source representations using a convolutional gated unit and reduces repetition.
- 1 Introduction: The paper provides an example showing repetition in a conventional attention-based seq2seq summary on Gigaword.The highlighted text in Table 1 marks the repeated content.
- 1 Introduction: Attention-based seq2seq models can suffer from repetition, semantic irrelevance, grammatical errors, and inadequate reflection of the source’s main idea.The cited example includes repeated words caused by attention to highly weighted source terms.
- 1 Introduction: The proposed global encoding framework uses a convolutional gated unit to filter encoder outputs according to global source context.Parameter sharing lets the gate refine each time-step representation using information from the full source context.
- 1 Introduction: The model is evaluated on LCSTS and Gigaword, where it outperforms state-of-the-art methods with ROUGE-2 F1 scores of 26.8 and 17.8, respectively.The passage reports results for both benchmark datasets.
- 1 Introduction: The analysis shows that global encoding reduces repetition compared with the conventional seq2seq model.This result is presented as an additional analysis of the proposed model.
2 Global Encoding
Global encoding refines encoder representations by combining convolutional features, self-attention, and a gate informed by the source context. The model retains the seq2seq attention framework while training generated summaries against reference summaries with negative log-likelihood.
- 2.1 Attention-based seq2seq: The encoder is bidirectional LSTM, and the decoder is unidirectional LSTM that generates summary words sequentially from a target vocabulary.The bidirectional encoder concatenates outputs from both directions at each time step.
- 2.2 Convolutional Gated Unit: A convolutional gated unit refines each encoder output using global source-context information before decoding.The model builds on an RNN encoder and attention-based seq2seq architecture.
- 2.2 Convolutional Gated Unit: Convolution extracts shared n-gram features, while self-attention captures long-term relationships among source annotations.The two modules jointly provide local feature patterns and global correlations.
- 2.2 Convolutional Gated Unit: A sigmoid gate filters the RNN source representations, suppressing dimensions near 0 and retaining dimensions near 1.The gate is computed from the CNN and self-attention outputs.
- 2.3 Training: Training minimizes the negative log-likelihood between generated and reference summaries, equivalent to maximizing their conditional probability.The objective is defined over model parameters, source sequence, and reference summary.
3 Experiment Setup
The experiments evaluate the model on LCSTS and Gigaword using ROUGE F1 scores and comparisons with dataset-specific baselines, including a conventional seq2seq implementation.
- 3.1 Datasets: LCSTS uses Chinese social-media text-summary pairs, whereas Gigaword pairs the first sentence of news articles with their corresponding headlines.
- 3.1 Datasets: The study evaluates the model on LCSTS and Gigaword using ROUGE-1, ROUGE-2, and ROUGE-L F1 scores.LCSTS contains more than 2.4 million training pairs, while Gigaword uses 3.8 million training pairs.
- 3.3 Baseline Models: The experiments compare the proposed model with baseline results reported in prior papers and with the authors’ implementation of conventional seq2seq.
- 3.3 Baseline Models: LCSTS baselines include RNN, RNN-context, CopyNet, SRB, and DRGD, while Gigaword baselines include ABS, ABS+, Feats, RAS-LSTM, RAS-Elman, SEASS, and DRGD.
4 Analysis
The analysis reports stronger ROUGE scores, improved semantic relevance in an example, and substantially less repetition than conventional seq2seq.
- 4.2 Discussion: The analysis includes an example comparing summaries from the proposed model, conventional seq2seq, and the reference.
- 4.1 Results: ROUGE-2 improves by 3.7 on LCSTS and 1.5 on Gigaword over conventional seq2seq.The model’s ROUGE advantages over baselines are reported on both datasets, with significant advantages on LCSTS.
- 4.2 Discussion: The model generates a more coherent and semantically relevant Starbucks summary by emphasizing the price gap rather than the country.The baseline omits the price information and repeats “China,” while the proposed summary focuses on the source text’s main idea.
- 4.2 Discussion: On Gigaword, duplicate 1-gram through 4-gram rates are significantly lower than conventional seq2seq and similar to the reference.The analysis measures repetition as the percentage of sentence-level duplicates.
5 Related Work
Prior work developed statistical, rule-based, and neural approaches to abstractive summarization, including attention-based seq2seq and mechanisms for filtering secondary information.
- 5 Related Work: Earlier summarization research used statistical and linguistic-rule-based methods before neural approaches became prominent.
- 5 Related Work: The paper also reports sentence-level duplicate percentages on Gigaword as part of its analysis of repetition.
- 5 Related Work: Neural abstractive summarization research builds on sequence-to-sequence models with attention and recurrent decoders.
- 5 Related Work: The paper’s related-work context includes selective gating and deep recurrent generative decoding as approaches to controlling or filtering information.
6 Conclusion
The proposed convolutional gated unit performs global source-side encoding to preserve core information and filter secondary information. Experiments on LCSTS and Gigaword report stronger baseline performance, reduced repetition, and greater robustness to varied input lengths than conventional seq2seq.
- 6 Conclusion: The convolutional gated unit performs global encoding that preserves core information and filters secondary information.
- 6 Conclusion: Experiments on LCSTS and Gigaword show that the model outperforms baselines and reduces repetition in generated summaries.
- 6 Conclusion: The model is reported to be more robust to inputs of different lengths than conventional seq2seq.