PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Pre-trained Checkpoints for Sequence Generation Tasks

Sascha Rothe, Shashi Narayan, Aliaksei Severyn

arXiv:1907.12461v2cs.CL

TL;DR

The paper examines whether pre-trained checkpoints can effectively warm-start sequence-to-sequence generation models. It finds that pre-trained encoders are essential, while shared encoder–decoder weights benefit most tasks and reduce memory use.

  • Problem

    It remains unclear how beneficial large text-pre-trained models are for warm-starting sequence-to-sequence generation models.

  • Method

    The paper conducts an extensive study of leveraging pre-trained checkpoints to initialize sequence-generation models, including both encoder and decoder components.

  • Results

    Pre-trained encoders are essential, and most tasks benefit from sharing encoder–decoder weights, which also decreases the memory footprint.

  • Takeaways & Limitations

    Combining RoBERTa and GPT-2 can achieve strong results when vocabularies are shared, although RoBERTa in both encoder and decoder performs better.

  • Takeaways & Limitations

    The model lags behind Bottom-Up and UniLM systems on abstractive summarization.

Abstract

from arXiv · show

Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on multiple benchmarks while saving significant amounts of compute time. So far the focus has been mainly on the Natural Language Understanding tasks. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We developed a Transformer-based sequence-to-sequence model that is compatible with publicly available pre-trained BERT, GPT-2 and RoBERTa checkpoints and conducted an extensive empirical study on the utility of initializing our model, both encoder and decoder, with these checkpoints. Our models result in new state-of-the-art results on Machine Translation, Text Summarization, Sentence Splitting, and Sentence Fusion.

1 Introduction

Pre-trained checkpoints had transformed NLU, but their value for sequence-to-sequence generation remained unclear. This paper develops a compatible Transformer model and empirically studies checkpoint initialization across generation tasks.

  • Motivation: Pre-trained checkpoints established strong baseline performance on widely used NLU benchmarks.Examples include GLUE and SQuAD.
  • Motivation: Public checkpoints and inference code reduce compute because warm-starting typically requires fewer fine-tuning steps.The paper describes savings of hundreds of TPU/GPU hours and orders of magnitude fewer fine-tuning steps.
  • Research gap: Evidence was limited on whether large pre-trained models benefit warm-started seq2seq generation, especially machine translation and summarization.BERT’s pre-training objective had been argued to be poorly suited to decoding text.
  • Approach: The authors develop a Transformer-based seq2seq model compatible with public BERT, GPT-2, and RoBERTa checkpoints.They investigate many ways to combine these checkpoints for encoder and decoder initialization.
  • Evaluation: Experiments cover sentence fusion, sentence splitting, WMT14 En↔De translation, and abstractive summarization.Summarization uses Gigaword, CNN/DailyMail, and BBC extreme.
  • Findings: The models improve over random initialization and report new state-of-the-art results across four sequence-generation tasks.The study also identifies the pre-trained encoder and encoder-decoder weight sharing as important design findings.

2 Models and Pre-trained Checkpoints

The model adapts encoder-only and decoder-only pre-trained architectures into a Transformer seq2seq system. It uses checkpoint-compatible layers, tokenization choices, and publicly available model resources.

  • Architecture: The seq2seq model uses Transformer layers in both encoder and decoder, extending the encoder with encoder-decoder attention.The encoder inherits BERT Transformer layer implementations, while the decoder supports autoregressive generation.
  • Pre-trained checkpoints: BERT supplies bidirectional masked-token representations, whereas GPT-2 supplies a decoder-only autoregressive language-modeling architecture.These differing pre-training setups motivate their use in different parts of the sequence-generation model.
  • Model configurations: Most experiments use 12-layer base checkpoints with hidden size 768, filter size 3072, and 12 attention heads.Larger checkpoints use 24 layers, hidden size 1024, filter size 4096, and 16 attention heads.
  • Tokenization and resources: The experiments tokenize text with WordPiece to match BERT’s pre-trained vocabulary and use publicly available BERT and GPT-2 checkpoints.RoBERTa vocabulary treatment is compatible with GPT-2 SentencePiece tokenization.
  • Parameter accounting: The model description tracks total, embedding, checkpoint-initialized, and randomly initialized parameters across initialization settings.BERT and GPT-2 embeddings contain 23M and 39M parameters, while encoder-decoder attention contains 26M.

3 Investigated Model Variants

The study compares Transformer variants that differ in which components are checkpoint-initialized, whether encoder and decoder weights are shared, and which vocabulary is used. It also evaluates larger checkpoints and a decoder-only setup.

  • Random initialization: RND2RND is a fully randomly initialized Transformer encoder-decoder baseline.
  • BERT-based variants: BERT2RND initializes the encoder and shared embeddings from BERT while leaving the decoder randomly initialized.
  • BERT-based variants: RND2BERT initializes the decoder from BERT and masks its bidirectional self-attention for autoregressive decoding.
  • BERT-based variants: BERT2BERT initializes both sides from BERT, with encoder-decoder attention remaining randomly initialized.
  • Shared-weight variants: BERTSHARE shares encoder and decoder parameters, reducing memory from 221M to 136M parameters.A layer-wise attention mechanism produced nearly identical numbers on most tasks.
  • RoBERTa and GPT-2 variants: ROBERTASHARE uses a shared encoder-decoder initialized from RoBERTa, while ROBERTA2GPT uses RoBERTa and GPT-2 with a shared vocabulary but separate embeddings.
  • Decoder-only variant: GPT treats the input as a conditioning prefix and warm-starts a decoder-only model from GPT-2.Its memory footprint is smaller than an encoder-decoder setup, at 125M parameters.
  • Checkpoint combinations: BERT2GPT combines separate BERT and GPT-2 checkpoints and vocabularies, whereas ROBERTA2GPT uses RoBERTa with GPT-2’s vocabulary.The study also reports 24-layer BERTSHARE and ROBERTASHARE models; a 24-layer GPT setup did not improve results and was excluded.

4 Experiments and Results

Across sentence fusion, sentence splitting, machine translation, and summarization, pre-trained initialization generally improves sequence-generation performance, with encoder initialization especially valuable. Shared encoder-decoder setups and checkpoint choice produce task-dependent gains, including state-of-the-art results.

  • Sentence Fusion: SARI 89.3 versus 86.9 shows that initializing the encoder substantially improves sentence fusion over random initialization.With larger models, ROBERTASHARE reaches SARI 90.3.
  • Split and Rephrase: Initializing the encoder improves WikiSplit performance most, while BERTSHARE outperforms the other initialization setups.Larger 24-layer models show slight over-fitting, prompting early stopping after about 100k steps.
  • Machine Translation: +4 points on En→De and +3.6 points on De→En demonstrate the benefit of multilingual BERT initialization for WMT14 newstest2014 translation.Encoder initialization is more beneficial than decoder initialization, while sharing encoder and decoder variables adds no further boost in machine translation.
  • Abstractive Summarization: +6.85 ROUGE-L on BBC extreme summarization exceeds the average RND2RND baseline improvement, whereas Gigaword gains average only +1.02 ROUGE-L.The larger gains on BBC are associated with the need for document-level inference and abstraction; BERTSHARE outperforms BERT2BERT on all three datasets.
  • Abstractive Summarization: ROBERTASHARE matches MASS on Gigaword, sets a new state of the art on BBC extreme summarization, and outperforms existing baselines on CNN/DailyMail.The proposed models use publicly available checkpoints and fine-tune only on the target task, while lagging behind Bottom-Up and UniLM on CNN/DailyMail.

5 Discussion on Ablation Studies

The ablations show that checkpoint compatibility, full-layer initialization, and vocabulary alignment strongly affect sequence-generation performance. Shared or fully initialized models generally outperform partial, mixed-checkpoint, or embedding-only alternatives.

  • Combining Different Checkpoints: Shared RoBERTa and GPT-2 checkpoints can help, but ROBERTA2GPT does not outperform using RoBERTa in both encoder and decoder.Combining BERT and GPT-2 often underperformed random initialization, whereas shared RoBERTa vocabulary enabled stronger results without surpassing the shared RoBERTa setup.
  • Initializing only Embeddings: 87.1 using BERT embeddings and 87.0 using GPT-2 embeddings barely improved over the 86.9 RND2RND baseline.Embedding-only initialization offered greater gains with just 10% of the training data, up to 2 percentage points.
  • Initializing only Layers: Removing embeddings from warm-started models caused a 1.7-point drop for BERTSHARE and an 11-point drop for GPT.Training new embeddings before fine-tuning improved BERTSHARE by 0.5 points but remained behind full initialization; GPT lacked a satisfying training curve.
  • Initializing a Subset of Layers: 89.1 SARI from selected large-BERT layers approached but did not exceed the 89.3 score of the base BERT checkpoint.The selected layers were 9, 10, 13-18, 23, and 24; the model retained comparable training speed and batch size.

6 Analysis of Abstractive Summaries

The analysis compares generated and reference extreme summaries using qualitative examples, automatic measures, and human judgments. RoBERTa-shared summaries were closest to gold quality, while repetition and factuality remained areas for improvement.

  • Human Assessment of Summary Quality: Human evaluation used Best-Worst Scaling to compare informativeness and fluency across four models and gold summaries.Scores were computed as the percentage chosen best minus the percentage chosen worst, ranging from -1 to 1.
  • Human Assessment of Summary Quality: ROBERTASHARE ranked best overall, followed by BERTSHARE, in human assessments of summary quality.Pairwise comparisons tested whether differences between systems were statistically significant.
  • Summary Lengths and Repetitions: ROBERTASHARE summaries were closest to GOLD in length, averaging 21.70 versus 24.61 words.All models produced summaries of comparable lengths, ranging from 20.71 to 21.70 words on average.
  • Summary Lengths and Repetitions: BERTSHARE and ROBERTASHARE reduced repetitions relative to RND2RND but retained more repetitions than RND2GPT.Rare or content-word repetitions were counted after removing the 500 most common words.
  • Qualitative Analysis: BERTSHARE and ROBERTASHARE were unequivocally better than RND2GPT in automatic ROUGE evaluations and human summary-quality evaluations.Examples show RND2RND repetition, RND2GPT paraphrasing, BERTSHARE repetition, and ROBERTASHARE output closer to the gold summary.

7 Related Work

Related work established pre-trained representations and models for NLU, while sequence-generation pre-training explored encoder-decoder initialization and task-specific objectives. Public checkpoint reuse remained an important direction, with limitations in task coverage and checkpoint availability.

  • Representation learning: Unsupervised pre-training evolved from static word embeddings to contextualized representations trained with large neural models.Word embeddings improved downstream tasks but were context-invariant; ELMo, ULMFiT, BERT, and GPT popularized contextualized pre-training.
  • Pre-training for sequence generation: Prior seq2seq pre-training initialized recurrent encoders and decoders, while MASS pre-trained encoder-decoder attention but produced a task-specific model.Ramachandran et al. reported gains on translation and summarization; MASS offered broader pre-training of the seq2seq mechanism but was less general than BERT or GPT-2.
  • Pre-training for sequence generation: UniLM unified bidirectional, unidirectional, and seq2seq language modeling, but no public checkpoint was available to the authors at writing.The paper compared against UniLM's reported results despite lacking access to its public checkpoint.
  • Pre-training models: BERT and GPT-based pre-trained models were primarily applied to classification, regression, question answering, summarization, and language modeling tasks.The related work emphasizes that many established applications were NLU tasks, while GPT-2 also covered several broader NLP tasks.
  • Leveraging public checkpoints: GPT-2 outperformed some unsupervised baselines on translation, reading comprehension, summarization, and language modeling, but remained behind supervised or task-specific approaches on the other tasks.GPT-2 achieved new state-of-the-art results on several language-modeling datasets, whereas its results on the other tasks were weaker relative to supervised methods.
  • Leveraging public checkpoints: The study excluded experiments with XLNet because its public checkpoint was released after most experiments had been completed.XLNet had achieved new state-of-the-art results on several NLP tasks, but was left for future work.

8 Conclusion

The study finds that public pre-trained checkpoints can substantially support sequence generation. Encoder initialization is essential, while encoder-decoder weight sharing and compatible vocabularies further affect results and efficiency.

  • 8 Conclusion: Pre-trained encoders are essential for sequence generation, and most tasks benefit from sharing weights between the encoder and decoder.Weight sharing also decreases the model's memory footprint.
  • 8 Conclusion: Combining BERT and GPT-2 often underperformed a randomly initialized baseline, whereas combining RoBERTa and GPT-2 achieved strong results.The comparison highlights the importance of sharing the vocabulary across the combined checkpoints.
  • 8 Conclusion: Training a language-specific BERT model improved performance over using the multilingual version.
Loading 1907.12461v2…