ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training

Motivation

• Autoregressive Seq2Seq models can capture local coherence while underfitting global coherence and long-term dependencies.Strong local correlations and one-step teacher forcing provide no explicit bias toward future-token planning.Source: §1 · p. 1

ProphetNet

• ProphetNet adds future n-gram prediction to traditional one-step-ahead language modeling.The additional objective predicts multiple future tokens and explicitly encourages future planning.Source: §1 · pp. 1–2

Architecture

• N-stream self-attention provides one main stream and n predicting streams for simultaneous future-token prediction.Each predicting stream attends to main-stream hidden states to predict a different future token.Source: §1 · p. 2
• Shared main-stream parameters allow ProphetNet to disable n-stream attention during inference and predict only the next token.This preserves the original Transformer Seq2Seq inference and fine-tuning behavior.Source: §1 · p. 2

Experiments

• ProphetNet uses mask-based auto-encoder denoising with 16GB and 160GB pre-training datasets before downstream fine-tuning.The experiments evaluate abstractive summarization and question generation tasks.Source: §1 · p. 2

2.2 Future N-gram Prediction

• ProphetNet introduces future n-gram prediction, replacing one-step prediction with simultaneous prediction of the next n continuous tokens at each time step.The objective combines the traditional language-modeling loss with n−1 future-token prediction losses.Source: §2.2 · pp. 2–3
• The future n-gram loss encourages future-token planning and reduces overfitting to strong local correlations.The loss is combined with the traditional language-modeling objective using weights α_j, with γ described as the attenuation coefficient.Source: §2.2 · p. 3

2.3 N-Stream Self-Attention

• Its decoder uses n extra predicting streams alongside the main Transformer self-attention stream, with the i-th stream modeling the i-th future token.The main stream preserves masked self-attention, while predicting streams use previous main-stream hidden states to predict future positions.Source: §2.3 · pp. 3–4
• Predicting-stream parameters are shared with the main stream, allowing all predicting streams to be disabled for conventional Transformer decoding during inference or fine-tuning.This design supports simultaneous future-token training without changing the next-token decoding form used after training.Source: §2.3 · p. 4

2.4 Seq2Seq Pre-training on Denoising Task

• For pre-training, ProphetNet masks token spans in an auto-encoder denoising task and recovers the next n future tokens within each masked span.The task follows token-span masking and differs from MASS by recovering multiple future tokens rather than one next token at each step.Source: §2.4 · pp. 4–5

Downstream tasks

• ProphetNet is fine-tuned for abstractive summarization on CNN/DailyMail and Gigaword, and for answer-aware question generation on SQuAD 1.1.The experiments use standard Seq2Seq task formulations and evaluate two SQuAD data splits.Source: §3.2 · p. 5, §3.3 · p. 6

Standard-scale pre-training

• ProphetNet achieves the best performance on all reported CNN/DailyMail ROUGE metrics.The evaluation reports F1 scores for ROUGE-1, ROUGE-2, and ROUGE-L on the test set.Source: §3.2 · p. 5
• ProphetNet outperforms previous models on all reported Gigaword metrics.Comparisons include OpenNMT, Re3Sum, MASS, and UniLM.Source: §3.2 · pp. 5–6

Question generation

• ProphetNet outperforms all previous methods on SQuAD 1.1 question generation with significant improvement.The same model and inference hyper-parameters are used for the two data splits with swapped development and test sets.Source: §3.3 · p. 6, Table 3 · p. 6

Large-scale pre-training

• 160GB pre-training yields state-of-the-art results on CNN/DailyMail and Gigaword, exceeding PEGASUS baselines on Gigaword with smaller pre-training corpora.ProphetNet uses about one-fifth and one-twentieth of the cited PEGASUS pre-training corpus sizes, respectively.Source: §3.4 · p. 7, Table 4 · p. 7

Without pre-training

• Without pre-training, the proposed method significantly improves performance over an architecture-matched original Transformer Seq2Seq model.The comparison uses the same architecture hyper-parameters for ProphetNet and the original Transformer.Source: §3.5 · p. 7

N-gram comparison

• ProphetNetbase-2gram performs comparably to ProphetNetbase-3gram while training 15% faster, motivating its use in subsequent experiments.Both variants outperform MASSbase and ProphetNetbase-1gram.Source: §3.6 · p. 8

• GPT uses left-to-right language modeling to predict subsequent tokens, whereas BERT and SpanBERT recover masked tokens or spans.Source: §4 · p. 8
• BART uses an encoder-decoder denoising objective that reconstructs original sentences from corrupted inputs.Source: §4 · p. 8
• Natural language generation methods typically generate one token per step using left-to-right or right-to-left language models.Source: §4 · p. 8
• Recent generation research incorporates future information through approaches including actor-critic modeling with a value function.Source: §4 · p. 8

• ProphetNet learns to predict future n-grams at each time step in sequence-to-sequence pre-training.Source: §5 · p. 8
• ProphetNet achieves the best performance on abstractive summarization and question generation tasks.Source: §5 · p. 8
• ProphetNet achieves new state-of-the-art results on CNN/DailyMail and Gigaword using about 1/3 the pre-training epochs of the previous model.Source: §5 · p. 8