Source-linked AI summary
Pretraining-Based Natural Language Generation for Text Summarization
Haoyu Zhang, Jianjun Xu, Ji Wang
TL;DR
Abstractive summarization methods lack complete decoding context and do not use pretrained contextualized language models on the decoder side. The paper proposes a BERT-based encoder-decoder with draft generation followed by masked-word refinement, achieving state-of-the-art results on CNN/Daily Mail and New York Times. It also reports that performance advantages narrow for summaries longer than 80 words.
Problem
Previous abstractive methods use left-context-only decoders and do not use pretrained contextualized language models on the decoder side.
Method
The model uses BERT in an encoder-decoder framework with draft generation, masked-word refinement, and a reinforcement objective.
Results
The model achieves state-of-the-art results on CNN/Daily Mail and New York Times, including a 33.33 average ROUGE score on CNN/Daily Mail.
Takeaways & Limitations
The two-stage model effectively improves performance over previous systems on the two evaluated summarization datasets.
Takeaways & Limitations
The model's advantage over the Lead-3 extractive baseline falls when golden summary length exceeds 80 words, possibly because long documents and summaries are truncated.
Abstract
from arXiv · showhide
In this paper, we propose a novel pretraining-based encoder-decoder framework, which can generate the output sequence based on the input sequence in a two-stage manner. For the encoder of our model, we encode the input sequence into context representations using BERT. For the decoder, there are two stages in our model, in the first stage, we use a Transformer-based decoder to generate a draft output sequence. In the second stage, we mask each word of the draft sequence and feed it to BERT, then by combining the input sequence and the draft representation generated by BERT, we use a Transformer-based decoder to predict the refined word for each masked position. To the best of our knowledge, our approach is the first method which applies the BERT into text generation tasks. As the first step in this direction, we evaluate our proposed method on the text summarization task. Experimental results show that our model achieves new state-of-the-art on both CNN/Daily Mail and New York Times datasets.
1 Introduction
The paper targets two limitations of abstractive summarization: left-context-only decoding and the absence of pretrained contextualized language models on the decoder side. It proposes a BERT-based, two-stage decoder and reports state-of-the-art results on CNN/Daily Mail and New York Times.
- Motivation: Abstractive summarization paraphrases and restructures source text to produce more flexible and diverse summaries.
- Challenges: Previous abstractive methods use only left context when predicting each word, leaving incomplete context for decoding.
- Challenges: These methods also omit pretrained contextualized language models on the decoder side, complicating summary representation, context-interaction, and language-model learning.
- Approach: Its two-stage decoder first generates a draft, then masks draft words and predicts refined words using bidirectional context; a reinforcement objective further improves naturalness.
- Approach: The proposed model uses BERT in the encoder and decoder and trains end-to-end without handcrafted features.
- Results: 33.33 average ROUGE-1, ROUGE-2 and ROUGE-L was achieved on CNN/Daily Mail, while New York Times ROUGE-1 improved by about 5.6% relative.
2 Background
The background frames summarization as a sequence-to-sequence task whose left-only decoding can produce unnatural outputs. It motivates bidirectional pretrained context encoders while noting their mismatch with incomplete decoder inputs.
- Text Summarization: Single-document multi-sentence summarization encodes a source sequence and predicts the target summary one word at a time.
- Text Summarization: Traditional decoders use only preceding context, although complete token context includes both preceding and following tokens.
- Text Summarization: Left-only decoding can degrade performance and produce unnatural sequences, including repeated phrases.
- Pretrained Context Encoders: Pretrained encoders such as ELMo, GPT, and BERT generate contextualized token embeddings from large unlabeled corpora.
- Pretrained Context Encoders: BERT uses multi-head self-attention and affine sublayers, with its last layer providing context encoding for the input sequence.
- Pretrained Context Encoders: BERT-based encoders and left-context-only decoders face a mismatch because pretraining uses complete bidirectional sequences, unlike incomplete decoder inputs.
3 Model
The model adds a word-level refine decoder to a BERT-based sequence-to-sequence framework and introduces a discrete objective to address exposure bias during generation.
- 3 Model: The model generates abstractive multi-sentence summaries from source documents within a BERT-based sequence-to-sequence framework.
- 3 Model: A word-level refine decoder addresses the two previously identified context and pretrained-language-model issues.
- 3 Model: A discrete objective is introduced for the refine decoders to reduce exposure bias.
3.1 Problem Formulation
Given a source document, the model first predicts a summary draft with a left-context-only decoder, then uses that draft to refine content with both context sides.
- 3.1 Problem Formulation: The input document is X = {x1, . . . , xm}, and the summary is Y = {y1, . . . , yL}, where L denotes summary length.
- 3.1 Problem Formulation: The model first predicts a draft summary from the input document using a left-context-only decoder.
- 3.1 Problem Formulation: It then conditions on both context sides to refine the draft, which guides and constrains the refinement process.
3.2 Summary Draft Generation
The summary draft generator encodes the document with BERT and uses a Transformer decoder to produce a draft summary autoregressively. A copy mechanism extends generation to source words outside the vocabulary.
- Encoder: BERT encodes input document tokens into representation vectors used by the summary draft decoder.The encoder output is denoted H, with each vector representing a source position.
- Draft decoder: The Transformer draft decoder predicts each summary word from previous draft outputs and encoder hidden representations.It uses decoder embeddings for prior outputs and encoder-decoder multi-head attention to learn source-summary alignments.
- Draft decoder: Using BERT directly in the draft decoder would expose it to incomplete contexts unlike its pretraining input distribution, harming contextual representations.The paper therefore does not use the BERT network to compute context vectors during draft decoding.
- Copy mechanism: A copy mechanism combines source-document copying with vocabulary generation to handle summary tokens that are out of vocabulary.The copying gate makes a soft choice between the two distributions and produces probabilities over the extended vocabulary V + X.
3.3 Summary Refine Process
The refine process improves decoder contextualization by masking draft words individually, encoding the partially masked draft with BERT, and predicting refined words with a Transformer decoder. It is designed to align decoding with BERT’s cloze-style pretraining and uses shared decoder parameters.
- Refine decoder: The refine decoder masks each draft word in turn, feeds the masked summary to BERT, and predicts the refined word using a Transformer decoder.At position t, the decoder uses the source document and the other summary words as context.
- Context modeling: The refine process gives BERT a more complete summary sequence and lets the decoder edit one word at a time using source and summary context.This addresses the incomplete-context problem of applying contextualized representations during draft generation.
- Context modeling: The word-level refine decoder mirrors BERT’s cloze pretraining task, which the authors associate with more fluent and natural generated sequences.The decoder predicts a masked position while the remaining summary words are visible.
- Parameter sharing: The draft and refine decoders share parameters because using separate parameters substantially degrades performance.The authors relate this choice to teacher-forcing and the refine objective’s similarity to language-model pretraining.
- Training objective: The model combines maximum-likelihood and discrete reward objectives for both draft and refine processes.The draft reward is based on ROUGE-L, while the mixed objective balances discrete optimization with readable sequence generation.
3.4 Learning and Inference
Training jointly optimizes the draft and refine losses with teacher forcing. At inference, beam search generates drafts and greedy search generates the refined summaries.
- Training: The model minimizes the sum of draft and refine objectives while jointly training both processes with teacher forcing.Ground-truth summaries are fed to each decoder during training.
- Inference: At test time, beam search generates draft summaries, while greedy search generates the refined summaries.Each decoder selects the predicted word at every time step according to its corresponding search procedure.
4 Experiment
Experiments evaluate the model on CNN/Daily Mail and NYT50 using ROUGE-based metrics, ablations, and summary-length analyses. The model improves over prior systems, while gains diminish for summaries longer than 80 words.
- CNN/Daily Mail evaluation uses full-length ROUGE-1, ROUGE-2 and ROUGE-L F1 scores, while NYT50 uses limited-length ROUGE recall.
- Ablation Analysis: 0.30 average ROUGE improvement over Two-Stage is obtained by adding the reinforcement objective, while removing refinement reduces average ROUGE by 1.69.
- Effects of Summary Length: For golden summaries of length 40–80, the model improves more over Pointer-Generator with Coverage than on shorter samples.
- Effects of Summary Length: The model’s advantage over Lead-3 declines when golden summary length exceeds 80, possibly because truncation loses information or training data are sparse.
5 Related work
Related work spans extractive and abstractive summarization, including salience selection, inconsistency control, and information selection. Pre-trained language models motivate this paper’s BERT-based two-stage abstractive model.
- Extractive methods improve performance through salience estimation, iterative sentence extraction, sentence compression, and latent-variable modeling.
- Abstractive methods use content selection, inconsistency loss, and information-selection networks to produce more informative summaries.
- Pre-trained language models such as ELMo, GPT, and BERT provide contextualized token embeddings after training on large unlabeled corpora.
- Prior summarization work uses pre-trained language models for sentiment analysis or decoder-side prior knowledge.
- This paper uses BERT on both encoder and decoder sides with a two-stage decoding structure for abstractive summarization.
6 Conclusion and Future Work
The paper proposes a BERT-based two-stage sequence-to-sequence model with a reinforcement objective and evaluates it on two benchmark datasets. The authors report improved performance over previous systems and suggest broader applicability to natural language generation tasks.
- The proposed model uses BERT on both encoder and decoder sides in a two-stage sequence-to-sequence architecture with a reinforcement objective.
- Experiments on CNN/Daily Mail and New York Times show improved performance compared with previous systems.
- The authors state that the model, refine decoder, and mixed objective can extend to machine translation, question generation, and paraphrasing.