Source-linked AI summary
Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, Weizhu Chen
TL;DR
Earlier text-generation methods relied mainly on sequential RNNs, which face long-term dependency and exposure-bias issues. GENIE combines an encoder with a diffusion decoder and pre-trains it using continuous paragraph denoising. Across four benchmarks, it achieves competitive performance with Transformer-based autoregressive methods and improves diversity.
Problem
Earlier text-generation methods mainly used sequential RNNs, which suffer from long-term dependency and exposure-bias issues.
Source: §1 · p. 1Method
GENIE uses an encoder-guided diffusion decoder and pre-trains it to predict noise added to continuous paragraphs from paragraph context and noisy-paragraph information.
Source: §1 · p. 1Results
Across four benchmarks, GENIE achieves competitive performance with Transformer-based autoregressive methods and shows significant improvements in diversity.
Source: §1 · pp. 1–2Takeaways & Limitations
Experiments across NLG tasks support the benefits of large-scale corpus pre-training for a diffusion model that generates high-quality, diverse text.
Source: §6 · p. 8
Abstract
from arXiv · showhide
In this paper, we introduce a novel dIffusion language modEl pre-training framework for text generation, which we call GENIE. GENIE is a large-scale pretrained diffusion language model that consists of an encoder and a diffusion-based decoder, which can generate text by gradually transforming a random noise sequence into a coherent text sequence. To pre-train GENIE on a large-scale language corpus, we design a new continuous paragraph denoise objective, which encourages the diffusion-decoder to reconstruct a clean text paragraph from a corrupted version, while preserving the semantic and syntactic coherence. We evaluate GENIE on four downstream text generation benchmarks, namely XSum, CNN/DailyMail, Gigaword, and CommonGen. Our experimental results show that GENIE achieves comparable performance with the state-of-the-art autoregressive models on these benchmarks, and generates more diverse text samples. The code and models of GENIE are available at https://github.com/microsoft/ProphetNet/tree/master/GENIE.
1. Introduction
The introduction presents GENIE as a diffusion-based text-generation approach with a continuous paragraph denoise pre-training objective. It situates the work against known limitations of sequential RNNs and the speed–precision tradeoff between autoregressive and non-autoregressive decoding, then reports benchmark results and analyses.
1. Introduction
- GENIE combines an encoder with a diffusion decoder that iteratively denoises output text from Gaussian noise, and is presented as a large-scale diffusion-pretrained model for sequence-to-sequence tasks.The introduction describes this as the first large-scale language pre-trained model based on the diffusion framework.Source: §1 · pp. 1–2
- Continuous paragraph denoise (CPD) trains GENIE to predict noise added to continuous paragraphs, supporting paragraph-level coherence and denoising.The objective uses paragraph context and noisy paragraph information to predict noise at the current diffusion step.Source: §1 · pp. 1–2
- Across XSum, CNN/DailyMail, Gigaword, and CommonGen, GENIE achieves competitive performance with Transformer-based autoregressive methods and shows improved diversity.The introduction also reports that the proposed pre-training method can improve performance.Source: §1 · pp. 1–2
- The evaluation includes an automatic annotation method for multiple outputs based on a large language model, alongside analyses of model behavior and properties.The introduction also notes ablations of diffusion steps and pre-training steps.Source: §1 · p. 2
2. Preliminary
The preliminary section frames sequence generation as conditional modeling from a source text to a target. It then describes diffusion as a forward noise-adding process and a learned reverse process.
2.1. Task Definition
- Sequence generation maps a source text to a target by modeling the conditional probability p(y | s).The task definition denotes the source as s and the target as y.Source: §2.1 · p. 2
2.2. Diffusion model
- The forward diffusion process adds Gaussian noise according to a variance schedule, with each next latent state determined by the current state; later states approach standard Gaussian noise.The process is described as a discrete-time Markov process.Source: §2.2 · p. 2
- The reverse process predicts noise from the current state to reconstruct the previous state, and the diffusion model is trained by minimizing a loss under the variational lower-bound approach.The passages describe a mean-squared error between predicted and true means, with the true mean computed from the reverse conditional distribution.Source: §2.2 · p. 2
3. Model
GENIE combines a bidirectional source encoder with a cross-attention diffusion model that iteratively turns Gaussian noise into text. Its continuous paragraph denoise objective pre-trains the model to predict noise in a paragraph using its surrounding context.
3. Model
- GENIE encodes the source bidirectionally, then iteratively denoises target embeddings in parallel, conditioning on the source representation through cross-attention.At inference, it starts from Gaussian noise and clamps the final continuous state to its nearest word embeddings before decoding tokens.Source: §3 · pp. 2–3
- Training minimizes both predicted-versus-true noise error and reconstruction error between the clean state and target embeddings.At a randomly sampled diffusion step, the denoiser predicts noise conditioned on the source representation.Source: §3 · p. 3
3.1. Pre-training GENIE
- Continuous paragraph denoise (CPD) masks a sampled paragraph in its document and trains the diffusion model to predict noise in that paragraph from its surrounding context.The paragraph is corrupted through forward diffusion; the CPD objective applies the sequence-to-sequence loss with the paragraph as target.Source: §3.1 · p. 4
- CPD is self-supervised and uses the original pre-training corpus without external labeled data, supporting semantic understanding and denoising at each diffusion step.The paper states that the task can fully exploit information in the original corpus.Source: §3.1 · p. 4
4. Experiments and Results
The experiments pre-train GENIE on a large corpus, fine-tune it for summarization and commonsense generation, and compare it with non-autoregressive, semi-autoregressive, and autoregressive baselines. The supplied passages describe the evaluation settings and table scopes but do not provide benchmark outcome values.
4.1. GENIE Pre-training
- Pre-training uses 160GB of news, books, stories, and web text, with input sequences capped at 512 tokens.The CPD target ratio is 30% (153 tokens for a 512-token input); training ran for 5 million steps over 50 days on eight 40GB NVIDIA A100 GPUs.Source: §4.1 · p. 4
4.2. Fine-tune on Downstream Tasks
- Fine-tuning evaluates summarization on GIGAWORD, CNN/DAILYMAIL, and XSUM for 120K steps each, reporting test-set ROUGE-1/2/L F1, and commonsense generation on COMMONGEN after 10K steps.CommonGen reports F1 scores for ROUGE-2/L, BLEU-3/4, CLDEr, and SPICE.Source: §4.2 · pp. 4–5
- The cited tables report Semi-NAR, NAR, and AR results on XSum, CNN/DailyMail, and Gigaword; Table 1 lists ROUGE-1 and ROUGE-2 as OVERALL average components before truncating.Source: Table 1 · p. 5, Table 2 · p. 5
4.3. Baselines
- The baseline set spans non-autoregressive and semi-autoregressive systems, plus autoregressive models including large-scale pretrained MASS, BART, T5, BANG, and ProphetNet.The pretrained baselines use base versions with parameter counts equivalent to GENIE.Source: §4.3 · p. 5
4.4. Main Results
GENIE performs competitively across the text-generation benchmarks, with strong reported comparisons on XSum and CommonGen. Its generated summaries are more diverse than BART’s, while LLM-based evaluation finds that many remain high quality.
4.4. Main Results
- GENIE outperforms other non-autoregressive and semi-non-autoregressive methods on XSum, matches pretrained autoregressive quality on three summarization datasets, and surpasses baselines including T5 on CommonGen.The CommonGen comparison is reported in Table 3.Source: §4.4 · pp. 5–6, Table 3 · p. 6
4.5. Generate Diversity Comparison
- GENIE greatly increases generation diversity over BART, whose diverse decoding and sampling methods yield only slight improvements; its outputs vary beyond a few words.The comparison uses SELF-BLEU scores for ten outputs per source.Source: §4.5 · p. 6, Table 4 · p. 7
- The case study reports that BART can produce high-quality single outputs but struggles to diversify multiple outputs, often repeating prefixes; GENIE maintains quality with richer diversity.Source: §4.5 · p. 6
- GENIE’s LLM-evaluated scores are slightly below BART’s, but the authors report enough high-quality summaries despite GENIE’s greater diversity.The authors characterize the score difference as acceptable given the diversity–quality trade-off.Source: §4.5 · p. 6, Table 5 · p. 7
4.6. Impact of Pre-training Steps
The experiments examine how pretraining duration, continuous-paragraph proportion, time-step sampling, and reverse-diffusion steps affect generation. They report improved quality with pretraining, a stable setting at γ = 30%, and degradation when too few reverse steps are used.
4.6. Impact of Pre-training Steps
- Pretraining for 1 million steps significantly improves XSum generation over training GENIE from scratch, and further pretraining steadily boosts downstream performance.Source: §4.6 · p. 7
4.7. Impact of Pre-training Parameters
- At γ = 30%, CPD pretraining is more stable and effective; proportions that are too high or too low lead to instability and poor performance.The comparison varies γ from 15% to 40% in 5% intervals.Source: §4.7 · p. 7
- Under 2 million pretraining steps, uniform time-step sampling outperforms loss-aware sampling at both tested γ values, 15% and 30%.The authors suggest uniform sampling lets the model learn sufficiently at each time step.Source: §4.7 · pp. 7–8
4.8. Impact of Diffusion Time Step
- With 2000 total diffusion steps, generation quality deteriorates at small reverse-step counts and stabilizes as the count increases to 1000.The tested reverse-step counts are 2000, 1000, 500, 250, and 100.Source: §4.8 · p. 8, Figure 2 · p. 8
5. Related Work
Related work spans large-scale language-model pretraining and diffusion-based text generation. GENIE is distinguished as a large-scale pretrained diffusion language model using a standard encoder–decoder framework.
5.1. Large Scale Pre-training Language Models
- Prior language-model pretraining includes left-to-right GPT models, bidirectional BERT and RoBERT models, and sequence-to-sequence tasks such as MASS and ProphetNet.MASS predicts masked spans step by step, while ProphetNet predicts multiple words per step.Source: §5.1 · p. 8
5.2. Diffusion Models for Text
- Earlier diffusion text methods include continuous-latent generation, text revision, and conditional generation; GENIE instead combines diffusion with a standard encoder–decoder and large-scale language-model pretraining.The paper describes GENIE as the first large-scale pretrained language model based on diffusion.Source: §5.2 · p. 8
6. Conclusion
The conclusion presents GENIE as a pretrained diffusion language model for sequence-to-sequence generation, pairing a bidirectional encoder with a non-autoregressive denoising decoder. The reported experiments support high-quality, diverse text generation and benefits from large-scale pretraining.
6. Conclusion
- GENIE uses a bidirectional source encoder and a non-autoregressive denoising decoder that iteratively refines noisy target text, with experiments reporting high-quality, diverse outputs and benefits from large-scale pretraining.Source: §6 · p. 8
A. Case Study
The case study contrasts GENIE’s varied summaries with BART’s repeated outputs, arguing that multiple perspectives can be useful even when a single GENIE summary may be less fluent. It presents diffusion generation as a way to broaden text-generation possibilities.
A. Case Study
- GENIE’s three summaries cover different source information, while BART’s outputs include near-duplicates despite diversity beam search.The examples describe distinct facets of the source in GENIE’s outputs and repeated descriptions in BART’s.Source: Table 9 · p. 11, A. Case Study · p. 12
- GENIE’s summaries may be less fluent than BART’s when judged one sentence at a time.The passage attributes this contrast to GENIE’s non-autoregressive generation mode.Source: A. Case Study · p. 12
- The case study argues that varied outputs can offer multiple perspectives, even when BART provides a concise summary.It notes that GENIE need not outperform autoregressive models in every respect.Source: A. Case Study · p. 12
- The authors present diffusion generation as a way to broaden text-generation approaches beyond autoregressive generation.They connect this prospect to varied content rather than errors or unrelated text.Source: A. Case Study · p. 12
B. Large Language Model Evaluation
The evaluation uses text-davinci-003 to score generated summaries. A fixed prompt combines the source article, evaluation instructions, and the summary, and the resulting scores are aggregated across ten generated samples.
B. Large Language Model Evaluation
- The authors use text-davinci-003 to evaluate generated summary quality.They identify prompt construction as especially important to this evaluation.Source: B. Large Language Model Evaluation · p. 12
- Each evaluation prompt contains the original article, evaluation requirements, and the summary being scored.The requirements remain the same for each summary.Source: B. Large Language Model Evaluation · p. 12
- Scores of 1, 2, and 3 represent bad, neutral, and good summaries; a high-quality summary is one scored 3.The authors aggregate average scores and the average number of high-quality summaries among ten generated samples.Source: B. Large Language Model Evaluation · p. 12
C. Impact of Sample Number
On XSum, the authors examine how the number of generated samples relates to similarity with the single reference answer. More samples increase the likelihood of a similar output, with the clearest improvement early in the increase, while diverse outputs complicate single-reference evaluation.
C. Impact of Sample Number
- GENIE’s diverse outputs are less suited to evaluation against a single reference answer.Different sampled Gaussian noises can produce different generations, even when guided by the source sequence.Source: C. Impact of Sample Number · p. 12
- On XSum, increasing the sample count from 5 to 10, 15, and 20 makes generated samples more likely to resemble the reference, with the clearest similarity improvement early.The figure compares results across those four sample counts.Source: C. Impact of Sample Number · p. 13, Figure 3 · p. 13
- More Gaussian noises sampled during inference increase the probability of approaching the single reference answer and raise the corresponding evaluation score.Source: C. Impact of Sample Number · p. 13