Source-linked AI summary
MASS: Masked Sequence to Sequence Pre-training for Language Generation
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu
TL;DR
Language-generation tasks are often low-resource or data-hungry, while methods such as BERT are designed primarily for language understanding. MASS addresses this gap by jointly pre-training an encoder-decoder model to reconstruct masked consecutive fragments, then fine-tuning it across multiple generation tasks. Across three tasks and eight datasets, it reports significant improvements and state-of-the-art unsupervised NMT performance, including more than four BLEU points over the previous state of the art on English-French.
Problem
Language-generation tasks are data-hungry and often low- or zero-resource, while directly applying BERT-like pre-training is not feasible because BERT targets language understanding.
Method
MASS jointly pre-trains an encoder-decoder model by reconstructing a masked consecutive sentence fragment from the remaining tokens, with additional masking of decoder inputs.
Results
MASS achieves significant improvements across three language-generation tasks and eight datasets, with state-of-the-art unsupervised NMT results and more than four BLEU points over the previous state of the art on English-French.
Takeaways & Limitations
A single MASS-pre-trained model can be fine-tuned for NMT, text summarization, and conversational response generation in low- or zero-resource settings.
Takeaways & Limitations
Future work is needed to apply MASS to additional sequence-generation tasks and to investigate its theoretical and empirical properties further.
Abstract
from arXiv · showhide
Pre-training and fine-tuning, e.g., BERT, have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks. Inspired by the success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks. MASS adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. In this way, MASS can jointly train the encoder and decoder to develop the capability of representation extraction and language modeling. By further fine-tuning on a variety of zero/low-resource language generation tasks, including neural machine translation, text summarization and conversational response generation (3 tasks and totally 8 datasets), MASS achieves significant improvements over the baselines without pre-training or with other pre-training methods. Specially, we achieve the state-of-the-art accuracy (37.5 in terms of BLEU score) on the unsupervised English-French translation, even beating the early attention-based supervised model.
1. Introduction
MASS addresses the mismatch between data-hungry, low-resource language generation and encoder-focused pre-training by jointly pre-training an encoder-decoder model to reconstruct masked sentence fragments. It is then fine-tuned across NMT, summarization, and conversational response generation, with reported gains including state-of-the-art unsupervised NMT results.
- Language generation tasks such as NMT, summarization, and conversational response generation are often data-hungry despite low- or zero-resource settings.
- MASS pre-trains an encoder-decoder model by encoding a sentence with a masked consecutive fragment and predicting that fragment in the decoder.
- MASS jointly trains encoder representation extraction and decoder language modeling by predicting masked fragments while masking decoder inputs that remain unmasked on the source side.
- One pre-trained model is fine-tuned on three language-generation tasks across eight datasets, including low-resource and unsupervised NMT settings.
- MASS achieves significant improvements over baselines and state-of-the-art unsupervised NMT results, including a BLEU score exceeding prior results by more than four points on English-French.
2. Related Work
Prior sequence-to-sequence pre-training methods offered limited or specialized support for language generation, while mainstream NLP pre-training emphasized language understanding. MASS instead jointly pre-trains both encoder and decoder components using unlabeled data for broad language-generation applications.
- Sequence-to-sequence learning supports language-generation applications including NMT, text summarization, question answering, and conversational response generation.
- Many language-generation tasks lack paired data but contain abundant unpaired data, motivating pre-training on unpaired data followed by fine-tuning with small paired datasets.
- Earlier NLP pre-training largely targeted language understanding through feature-based or fine-tuning approaches.
- Earlier encoder-decoder pre-training methods showed limited and less general improvements, while some approaches pre-trained only the encoder.
- MASS differs by jointly pre-training the encoder and decoder using only unlabeled data and applying the method to most language-generation tasks.
3. MASS
MASS pre-trains an encoder-decoder model by reconstructing a masked consecutive sentence fragment from the remaining tokens. Its masking design jointly develops encoder representations and decoder language modeling, while encompassing BERT-style masked language modeling and standard language modeling as limiting cases.
- Masked Sequence to Sequence Pre-training: The encoder-decoder framework encodes the unmasked sequence, and the decoder predicts the masked fragment using its representations.The decoder receives masked inputs at positions outside the target fragment in the illustrated training setup.
- Masked Sequence to Sequence Pre-training: MASS predicts a masked sentence fragment from the corresponding sentence with that fragment replaced by mask symbols.The masked fragment contains consecutive tokens, while the sentence length remains unchanged.
- Discussions: The masked-fragment length k determines the pre-training regime, with k = 1 corresponding to BERT-style masked language modeling and k = m to standard language modeling.At k = 1 the decoder predicts one masked token from unmasked source tokens; at k = m the encoder provides null information and the decoder predicts the full sentence autoregressively.
- Discussions: MASS is designed for encoder-decoder language generation rather than solely encoder- or decoder-focused pre-training.The paper applies the sequence-to-sequence framework to language generation, using Transformer in its experiments.
- Masked Sequence to Sequence Pre-training: MASS jointly pre-trains encoder and decoder capabilities by requiring semantic extraction from unmasked tokens and autoregressive prediction of consecutive masked tokens.Masking decoder inputs that are unmasked on the encoder side encourages reliance on encoder representations rather than previous-token information alone.
4. Experiments and Results
MASS is evaluated after pre-training on monolingual data and fine-tuning across unsupervised and low-resource language-generation tasks. It consistently improves over non-pretrained and alternative-pretraining baselines across NMT, summarization, and conversational response generation.
- Experimental Setup: MASS is pre-trained on WMT monolingual corpora and fine-tuned across NMT, text summarization, and conversational response generation.The experiments include low-resource settings and unsupervised NMT, using Transformer encoder-decoder models.
- Unsupervised NMT: 4.1 BLEU points: MASS outperforms XLM on unsupervised English-French translation and exceeds previous methods on all six translation directions.The comparison includes methods without pre-training and with BERT-like pre-training.
- Unsupervised NMT: MASS outperforms both BERT+LM and DAE on all unsupervised translation pairs.The paper attributes the advantage to jointly training the encoder and decoder so the decoder cannot simply copy unmasked input tokens.
- Text Summarization: More than 10 ROUGE points on 10K data and 5 ROUGE points on 100K data: MASS improves over the no-pre-training baseline in low-resource summarization.The gains are reported across different scales of Gigaword fine-tuning data.
- Conversational Response Generation: MASS achieves lower PPL than the baseline and other pre-training methods on both 10K and 110K conversational-response training data.The method also performs best when the masked fragment is approximately 50% of the sentence length, while discrete masking and feeding all decoder tokens perform worse.
5. Conclusion
MASS improves language generation across three tasks and eight datasets, including unsupervised NMT, while extending the same pre-trained model through fine-tuning. The authors identify broader task coverage and further analysis as future work.
- MASS achieved significant improvements over baselines without pre-training or with other pre-training methods across three language generation tasks and eight datasets.The tasks include neural machine translation, text summarization, and conversational response generation.
- MASS achieved state-of-the-art BLEU scores for unsupervised NMT on three language pairs, exceeding the previous state of the art by more than 4 BLEU points on English-French.
- Future work will apply MASS to sentence paraphrasing, text style transfer, post-editing, and other sequence generation tasks.The authors also plan further theoretical and empirical analysis of the method.