Pre-trained Language Model Representations for Language Generation

• The study examines representation-transfer strategies for neural machine translation and abstractive summarization, extending prior work focused mainly on language understanding.Source: §1 · p. 1
• Encoder inputs provide the most successful integration strategy for pre-trained representations in sequence-to-sequence language generation.The experiments use transformer language models trained on up to six billion tokens.Source: §1 · p. 1
• Pre-trained representations improve neural machine translation most in low-resource settings, but gains over 1 BLEU remain possible with 5.2M sentence-pairs.Source: §1 · p. 1

2.1 ELMo augmentation

• ELMo augmentation combines normalized language-model layer outputs and learned weights into contextualized word embeddings without changing language-model parameters.The module also incorporates the language model’s input word embeddings and applies layer normalization to each layer output.Source: §2.1 · pp. 1–2
• ELMo representations are supplied either to the encoder or decoder, providing input-specific word representations trained on more data than the generation task.Source: §2.1 · p. 2

2.2 Fine-tuning approach

• Fine-tuning replaces encoder input embeddings with a pre-trained language-model representation and updates the language model using the end-task learning signal.The representation comes from the layer before the softmax, with dropout applied to the language-model output.Source: §2.2 · p. 2
• The same fine-tuning strategy can add language-model representations to the decoder while jointly training the sequence-to-sequence model.Source: §2.2 · p. 2

3.2 Language model pre-training

• Language models are trained on German and English newscrawl containing up to 6B and 5B tokens, respectively, with a joint 37K-type BPE vocabulary.Source: §3.1 · p. 2

3.1 Datasets

• The study evaluates WMT’18 English-German translation on 5.18M filtered sentence-pairs and WMT’18 English-Turkish translation on 208K unfiltered sentence-pairs.Both translation tasks use case-sensitive detokenized BLEU.Source: §3.1 · p. 2
• CNN-DailyMail contains over 280K news articles with multi-sentence summaries, using non-anonymized articles truncated to 400 tokens and evaluated with F1-Rouge.Source: §3.1 · p. 2

3.2 Language model pre-training

• The pre-trained models include bidirectional and unidirectional self-attention architectures, each with 12 transformer blocks and 353M or 190M parameters.Source: §3.2 · pp. 2–3

3.3 Sequence to sequence model

• Translation uses Big Transformer models with six encoder and decoder blocks, while summarization uses a base Transformer model optimized with tuned dropout and beam search.Source: §3.3 · p. 3

4.1 Machine translation

• Pre-trained representations are most effective in low-bitext settings, while SRC-ELMO remains beneficial with 5.2M sentence pairs.SHARED gains erode to practically zero in large-bitext settings, whereas SRC-ELMO still improves the 5.2M setting by over 1 BLEU.Source: §4.1 · pp. 3–4
• 5.3 BLEU is gained over the baseline with SRC-ELMO+SHDEMB in the 160K bitext setup.SRC-ELMO alone improves the 160K setting by 3.8 BLEU and the 5.2M setting by over 1 BLEU.Source: §4.1 · p. 4

4.2 Abstractive summarization

• ELMo inputs achieve a new state of the art on abstractive summarization using CNN-DailyMail.The experiments use the non-anonymized, full-text version and compare against a strong Transformer baseline and a task-specific architecture.Source: Table 3 · p. 4, §4.2 · p. 4

4.1 Machine translation

• Encoder-side representations slow inference by only 12-14%, because language-model computation can be parallelized across input tokens.Decoder-side representations are much slower because the language model must be invoked repeatedly.Source: §4.1 · p. 4
• ELMo inputs to the encoder improve WMT’18 English-Turkish translation with 208K sentence-pairs.The result is reported in Table 2 for newstest2017 validation and newstest2018 testing.Source: §4.1 · p. 4, Table 2 · p. 4

• Pre-trained representations are most effective when added to the encoder network for neural machine translation and abstractive summarization.Source: §5 · p. 5
• Improvements diminish as labeled data increases but remain observable when millions of examples are available.Source: §5 · p. 5
• Future research will investigate ways to improve the decoder with pre-trained representations.Source: §5 · p. 5

• BLEU is reported on newstest2012 through newstest2018 for WMT English-German translation.Source: Table 4 · p. 7
• The table compares performance across various simulated bitext-size scenarios.Source: Table 4 · p. 7
• The results concern English-German translation under the experimental settings associated with Figure 1.Source: Table 4 · p. 7

• Training and inference speed are both measured for models trained on WMT English-German.Source: Table 5 · p. 8
• Training speed is measured using 32 V100 GPUs.Source: Table 5 · p. 8
• Inference speed is measured on a single V100 while batching up to 12K source or target tokens.Source: Table 5 · p. 8