Source-linked AI summary
ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
Dongling Xiao, Han Zhang, Yukun Li, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang
TL;DR
Existing NLG pre-training gives little attention to exposure bias between teacher-forced training and inference. ERNIE-GEN combines infilling and noise-aware generation with span-by-span generation and multi-granularity target sampling. It reports state-of-the-art results across NLG tasks with less pre-training data and parameters.
Problem
Current NLG pre-training works pay little attention to exposure bias, arising when training uses ground-truth words but inference uses generated words.
Source: §1 · p. 1Method
ERNIE-GEN uses multi-flow seq2seq pre-training and fine-tuning with infilling, noise-aware generation, span-by-span prediction, and multi-granularity target sampling.
Source: §1 · pp. 1–2Results
ERNIE-GEN achieves state-of-the-art results across several NLG tasks while using much less pre-training data and fewer parameters.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Current pre-training works in natural language generation pay little attention to the problem of exposure bias on downstream tasks. To address this issue, we propose an enhanced multi-flow sequence to sequence pre-training and fine-tuning framework named ERNIE-GEN, which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder. Experimental results demonstrate that ERNIE-GEN achieves state-of-the-art results with a much smaller amount of pre-training data and parameters on a range of language generation tasks, including abstractive summarization (Gigaword and CNN/DailyMail), question generation (SQuAD), dialogue generation (Persona-Chat) and generative question answering (CoQA).
1 Introduction
ERNIE-GEN addresses exposure bias in seq2seq generation with infilling and noise-aware training, and adds span-by-span generation and multi-granularity target sampling. It reports state-of-the-art results across several NLG tasks with less pre-training data and parameters.
1 Introduction
- Existing NLG pre-training pays little attention to exposure bias: training uses ground-truth words, but inference uses generated words, where errors tend to accumulate.Source: §1 · p. 1
- Its Transformer-based Multi-Flow Attention uses infilling to gather history at training and inference, while target-word corruption trains mistake detection and ignoring.Infilling uses an inserted [ATTN] symbol and its position; corrupted targets are created by randomly replacing words with arbitrary vocabulary words.Source: §1 · pp. 1–2
- Span-by-span generation predicts semantically complete spans consecutively rather than word by word, alongside an infilling-based word-by-word flow to facilitate convergence.Source: §1 · p. 2
- Multi-granularity target-fragment sampling encourages decoder reliance on encoder representations over previous generated words, enhancing encoder–decoder correlation.The passage contrasts this with methods that sample a single continuous target segment, particularly for long-text pre-training.Source: §1 · p. 2
- ERNIE-GEN achieves state-of-the-art results across summarization, question generation, dialogue generation, and generative question answering with much less pre-training data and parameters.Source: §1 · p. 2
2 Related Work
Related work covers pre-training for NLU and NLG, approaches to exposure bias, and span-level prediction. ERNIE-GEN's span-by-span task draws on span-prediction work while targeting generation.
2 Related Work
- ELMo [Peters et al., 2018], GPT, and BERT [Devlin et al., 2019] pre-train language models, but single-encoder or single-decoder designs are less effective for encoder-decoder generation tasks.GPT [Radford et al., 2018] learns a forward language model; BERT uses masked language modeling, while ELMo pre-trains forward and backward language models.Source: §2 · p. 2
- MASS predicts a masked consecutive input fragment, while UNILM masks words across paired encoder–decoder segments and predicts the masked words.Source: §2 · p. 2
- Variational autoencoders [Yang et al., 2019a; Wang et al., 2019] and reinforcement learning [Ranzato et al., 2016] address exposure bias, but are described as inefficient for large-scale pre-training.The passage attributes VAE inefficiency to inference for both posterior and prior distributions, and reinforcement-learning inefficiency to word-by-word sampling.Source: §2 · p. 2
- Span-prediction studies [Sun et al., 2019; Sun et al., 2020; Joshi et al., 2019] report better NLU performance; ERNIE-GEN proposes semantically complete span-by-span generation.Source: §2 · p. 2
3 Proposed Framework
ERNIE-GEN builds on infilling generation and trains word-by-word and span-by-span flows in parallel. Its framework combines multi-granularity fragment sampling, target noising, shared-flow attention, and inference-time infilling.
3 Proposed Framework
- The proposed framework uses an infilling generation mechanism to train the two generation flows in parallel.Source: §3 · p. 2
3.1 Multi-Granularity Target Fragments
- The model samples fragments from length distributions until a fragment budget is reached, removes and stitches them into target T, then predicts T from the remaining input.The selected setting uses a fragment-length budget of 25% of the input.Source: §3.1 · pp. 2–3
3.2 Noise-Aware Generation
- For noise-aware generation, ground-truth target words are randomly replaced to create a corrupted target, with separate noising rates for pre-training and fine-tuning.The rates are denoted ρp and ρf, respectively.Source: §3.2 · p. 3
3.3 Architecture: Multi-Flow Attention
- ERNIE-GEN trains word-by-word and span-by-span generation in parallel using a shared contextual flow; span candidates are selected from semantically informative n-grams.The span flow predicts each whole span from the same preceding history context.Source: §3.3 · pp. 3–4
3.4 Inference: Infilling Decoding
- At inference, [ATTN] is inserted step by step to gather history context, then dropped at each step for efficiency.Source: §3.4 · p. 4
4 Experiments
Experiments compare ERNIE-GEN with previous methods on language-generation tasks and use ablations to assess its proposed methods.
4.1 Pre-training and Implementation
- Pre-training uses English Wikipedia and BookCorpus (16GB), with base and large models initialized from BERTBASE and BERTLARGE, respectively.The base model has 110M parameters and trains for 400,000 steps; the large model has 340M parameters and trains for 450,000 steps.Source: §4.1 · p. 4, Figure 4 · p. 4
4.2 Fine-tuning on Downstream Tasks
- On Gigaword, ERNIE-GENLARGE gains +1.94 ROUGE-L over UNILMLARGE at 10k, while ERNIE-GENBASE beats PEGASUS using 110M parameters and 16G training data.The full-training-set comparison is reported as state of the art.Source: Table 2 · p. 4, §4.2 · p. 5
- On CNN/DailyMail, ERNIE-GENBASE beats MASS by +0.67 ROUGE-L and ERNIE-GENLARGE beats UNILMLARGE by +0.73 ROUGE-L.ERNIE-GENLARGE also achieves state-of-the-art ROUGE-L and comparable ROUGE-1/2 performance.Source: §4.2 · p. 5, Table 3 · p. 5
- On SQuAD question generation, ERNIE-GEN achieves a new state-of-the-art result, outperforming UNILMLARGE by +1.82 BLEU-4.The reported evaluation uses two different test–development split methods.Source: §4.2 · p. 5, Table 4 · p. 5
- On Persona-Chat, ERNIE-GEN outperforms the current task-specific pre-training model; on CoQA, it beats early works by +2.0 F1.The CoQA result is reported for generative question answering.Source: §4.2 · p. 5, Table 5 · p. 5, Table 6 · p. 5
4.3 Ablation Studies
- Without fine-tuning noising, infilling outperforms typical generation across tasks; noise-aware fine-tuning improves both variants, and appropriate noising benefits training.As noising increases, the model assigns less attention to noised target words, helping it detect and ignore false predictions during inference.Source: §4.3 · pp. 5–6, Table 8 · p. 6, Figure 5 · p. 6
- Adding noise-aware generation and span-by-span training improves performance across all three tasks over word-by-word infilling; noise-aware generation also beats masked-language-model fine-tuning.The comparison reports that the noise-aware method predicts all decoder-side words, unlike masking that learns only masked positions.Source: §4.3 · p. 6, Table 7 · p. 6
5 Conclusions
ERNIE-GEN combines infilling and noise-aware generation to address exposure bias, adds span-by-span generation, and achieves state-of-the-art results across NLG tasks.
5 Conclusions
- The framework combines infilling generation and noise-aware generation to alleviate exposure bias.It also adds span-by-span generation to train models to generate text more like human writing.Source: §5 · p. 6
- Span-by-span generation further improves downstream performance.Source: §5 · p. 6
- Experiments report state-of-the-art results across a range of NLG tasks.Future work includes reinforcement learning in pre-training and applying ERNIE-GEN to more tasks, including machine translation.Source: §5 · p. 6
A.1 Pre-training on Large-scale Text Corpora
ERNIE-GEN is evaluated after pre-training on a 430GB text corpus, with larger-scale pre-training reported to improve summarization performance significantly and SQuAD question-generation scores slightly.
A.1 Pre-training on Large-scale Text Corpora
- Larger-scale pre-training significantly increases performance on Gigaword and CNN/DailyMail.The evaluation is reported in Table 9; ERNIE-GEN‡LARGE was pre-trained on 430GB of text.Source: §A.1 · p. 8, Table 9 · p. 8
- ERNIE-GENLARGE was pre-trained on 430GB of text for one epoch and 1M training steps.The text corpus was extracted from corpora used by RoBERTa, T5, and ALBERT.Source: §A.1 · p. 8
- Larger-scale pre-training slightly improves Rouge-L and BLEU-4 on SQuAD question generation.Table 10 identifies ‡-marked ERNIE-GEN as pre-trained on the 430GB corpus.Source: §A.1 · p. 8, Table 10 · p. 8
- Table 11 presents fine-tuning hyperparameters for CNN/DailyMail, Gigaword, and SQuAD question generation.Source: §A.1 · p. 8, Table 11 · p. 8