Source-linked AI summary
Unsupervised Pretraining for Sequence to Sequence Learning
Prajit Ramachandran, Peter J. Liu, Quoc V. Le
TL;DR
Seq2seq models can overfit when supervised data are limited, motivating a method that initializes encoder and decoder networks from pretrained language models before labeled fine-tuning. Across translation and abstractive summarization, the method improves generalization, including a 1.3 BLEU gain on both WMT’14 and WMT’15 English→German and better human-rated summaries.
Problem
Seq2seq models can easily overfit when the amount of supervised training data is small.
Method
The method initializes both seq2seq encoder and decoder networks with pretrained language-model weights and fine-tunes them with labeled data while jointly training language-model objectives.
Results
The method improves generalization on translation and summarization, including a 1.3 BLEU improvement over previous best models on both WMT’14 and WMT’15 English→German.
Takeaways & Limitations
The method can aid both generalization and optimization and is flexible across a large variety of tasks.
Takeaways & Limitations
The reported non-anonymized summarization evaluation used a unidirectional LSTM with a 600-token context because of GPU memory issues, whereas a comparison used 800 tokens and a bidirectional LSTM.
Abstract
from arXiv · showhide
This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the art results on the WMT English$\rightarrow$German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves a significant improvement of 1.3 BLEU from the previous best models on both WMT'14 and WMT'15 English$\rightarrow$German. We also conduct human evaluations on abstractive summarization and find that our method outperforms a purely supervised learning baseline in a statistically significant manner.
1 Introduction
Seq2seq models can overfit when labeled data are scarce, so this work uses unsupervised language-model pretraining to improve supervised seq2seq learning. The method improves generalization and achieves strong results in translation and summarization.
- Small supervised datasets can cause seq2seq models to overfit.
- The method initializes both encoder and decoder networks with pretrained language-model weights, then fine-tunes them with labeled data and language-model objectives.Joint language-model training is used during fine-tuning to prevent overfitting.
- 1.3 BLEU improvement over previous best models is reported on both WMT’14 and WMT’15 English→German.
- Human evaluations find the pretrained model outperforms a purely supervised summarization baseline in correctness and avoiding unwanted repetition.
- Ablation studies report improved test-set generalization on both translation and summarization tasks.The study attributes translation gains mainly to pretrained features and summarization gains partly to improved encoder optimization.
2 Methods
The approach pretrains seq2seq components as language models, transfers selected weights into a multi-layer model, and jointly retains language-model objectives during supervised fine-tuning. Residual connections and multi-layer attention provide additional minor improvements.
- 2.1 Basic Procedure: Seq2seq learning maximizes the likelihood of an output sequence conditioned on an input sequence.
- 2.1 Basic Procedure: Because encoder and decoder components resemble language models, both seq2seq networks can be initialized with language-model parameters.
- 2.1 Basic Procedure: The procedure pretrains encoder and decoder networks on large unlabeled text datasets before supervised fine-tuning.
- 2.1 Basic Procedure: The embedding and first LSTM layers of both components, plus the decoder softmax, receive pretrained weights from source- and target-side language models.
- 2.2 Monolingual language modeling losses: Fine-tuning can cause catastrophic forgetting, so monolingual language-model losses regularize pretrained parameters alongside the seq2seq objective.The two losses are weighted equally.
- 2.3 Other improvements to the model: Residual connections and multi-layer attention yield minor but consistent improvements beyond pretraining and language-model losses.
- 2.3 Other improvements to the model: A residual connection feeds the first decoder LSTM layer directly into the softmax input to avoid random gradients reaching pretrained parameters.
- 2.3 Other improvements to the model: Multi-layer attention attends to encoder states from both the first and top layers.
3 Experiments
The experiments evaluate unsupervised pretraining for English→German translation and CNN/Daily Mail summarization, including benchmark comparisons, ablations, and data-size analyses. Results show gains from pretrained components and language-model objectives, with benefits especially pronounced when labeled translation data is limited.
- Experimental setup: The method is evaluated on WMT English→German translation and low-resource CNN/Daily Mail abstractive summarization.The experiments compare against prior systems and analyze individual components through ablation studies.
- Machine translation: The pretrained translation model achieves state-of-the-art single-model performance on WMT’14 and WMT’15 English→German.It significantly outperforms competitive semi-supervised backtranslation and surpasses the previous state-of-the-art four-model ensemble; a five-model ensemble matches or exceeds the previous twelve-model ensemble.
- Machine translation: Only pretraining the decoder loses 1.0 BLEU, whereas only pretraining the encoder loses 1.6 BLEU, making decoder-only pretraining the stronger single-side choice for translation.The ablation study compares each variant with the full model, whose encoder and decoder are initialized from monolingual language models and retain the language-modeling objective.
- Machine translation: The language-modeling objective is a strong regularizer, and pretraining the softmax is important for preserving translation performance.Removing the language-modeling objective performs as poorly as using it without pretraining, while omitting softmax pretraining causes a 1.6 BLEU-point drop.
- Machine translation: The pretrained translation model retains a larger advantage as labeled data decreases: the gap grows from 2.0 BLEU with all data to 3.8 BLEU with 20%.The pretrained models degrade less as the labeled dataset becomes smaller.
- Abstractive summarization: For summarization, encoder pretraining gives large improvements, while removing the language-modeling objective causes a significant ROUGE drop.The study interprets encoder pretraining as improving optimization, potentially by allowing gradients to flow further back through the unrolled encoder.
4 Related Work
Prior unsupervised pretraining improved other neural models but had rarely been applied to seq2seq learning. Related approaches include transfer, multitask learning, output scoring, and backtranslation, with differing scope and integration strategies.
- Unsupervised pretraining: Earlier unsupervised pretraining improved feedforward acoustic models, whereas later work found it unnecessary in some acoustic-modeling settings.The paper contrasts these findings with its own language-model pretraining results on seq2seq tasks.
- Related seq2seq methods: Prior semi-supervised pretraining methods lacked decoder networks and therefore did not apply directly to sequence-to-sequence learning.
- Related seq2seq methods: Other related approaches used source-side sentence reordering, transfer or multitask learning, or additional output layers combined with an independently trained language model.The paper reports only small improvements over a supervised baseline for the additional-output-layer approach.
- Theoretical connections: Chen et al.'s framework combines output-likelihood scoring with input dependence, while this work folds the pretrained language model into the decoder and uses labeled examples for the input-dependent term.
- Unsupervised machine translation: Backtranslation creates extra labeled data from unlabeled inputs and is described as complementary to the paper's method, with potential for additional gains.
5 Conclusion
The paper presents unsupervised pretraining for seq2seq learning, using language-model weights to improve both generalization and optimization across tasks.
- The method pretrains source and target language models, then initializes seq2seq embeddings, first LSTM layers, and softmax with their weights.
- State-of-the-art machine translation results were achieved on both WMT’14 and WMT’15 English to German.
- The technique is flexible and can be applied to a large variety of tasks.
A Example outputs
Example outputs generally show the pretrained model producing more relevant, informative, or structurally accurate outputs, while also exhibiting occasional errors and failure cases.
- The pretrained model produces a highly informative summary, whereas the no pretrain model outputs irrelevant details.
- The pretrained model generates a highly relevant summary but misidentifies the feline executioner’s name, while the baseline repeats irrelevant details.
- Both models produce relevant summaries, but the no pretrain model uses one name for both players.
- In one failure case, the pretrained model outputs irrelevant details while the no pretrain model successfully summarizes the document.
- The pretrained model avoids the baseline’s mistranslation and sentence-structure errors in translating a difficult German phrase.
- The pretrained model better captures meaning and sentence structure, but incorrectly translates 31.16% as 316%.
- Both models produce poor translations for poorly worded English, and both mistranslate “radical” as “radical Islam.”
- Both models preserve structure but miss “total vote count,” and human evaluation finds them equally poor.