Self-conditioned Embedding Diffusion for Text Generation

1 INTRODUCTION

• Diffusion predicts sequence tokens in parallel, allowing bidirectional attention and potentially more coherent samples than causal attention.The passage also notes that sequence-parallel computation can make better use of accelerator hardware at inference.Source: §1 · p. 1
• Prior text-diffusion attempts mostly used discrete approaches that cannot benefit from continuous-diffusion refinements such as guidance.The passage says guidance can drastically improve diffusion sample quality.Source: §1 · p. 1
• SED combines continuous diffusion in token-embedding space with self-conditioning to model text, addressing the mismatch between continuous diffusion and discrete tokens.The method uses a fixed embedding space, either random or from a trained language model.Source: §1 · p. 1
• SED models rival mainstream autoregressive models in both conditional and unconditional text generation.The paper reports comparable human-rated samples, though they are slightly worse, and a better likelihood-entropy trade-off for similar model sizes.Source: §1 · p. 2

Continuous diffusion on continuous image data

• Continuous diffusion advanced image generation, with later work introducing latent-space modeling and classifier-free guidance.The discussion traces these developments from Sohl-Dickstein et al. (2015) through Stable Diffusion (Rombach et al., 2021), GLIDE (Nichol et al., 2022), DALL-E 2 (Ramesh et al., 2022), and Imagen (Saharia et al., 2022).Source: §2 · p. 2

Discrete diffusion on discrete data

• Discrete diffusion and masked-token reconstruction approaches were developed for discrete data, but text samples from discrete diffusion remained less coherent than autoregressive samples.Examples include multinomial text modeling (Hoogeboom et al., 2021), D3PM (Austin et al., 2021), CMLM (Ghazvininejad et al., 2019), and SUNDAE (Savinov et al., 2022).Source: §2 · p. 2

Continuous diffusion on discrete data

• Continuous representations had shown promise for discrete music and constrained or low-data text, but had not been shown to scale to large realistic text datasets or match autoregressive models on generic language tasks.Diffusion-LM (Li et al., 2022) learns token embeddings jointly with the diffusion model; Analog Bits (Chen et al., 2022) applies self-conditioning to discretized images.Source: §2 · p. 2

Auto-regressive modelling on discrete data

• Autoregressive models remained the method of choice for discrete data, progressing from neural sequence models to Transformers and large language models.The cited trajectory includes Bengio et al. (2000), Sutskever et al. (2011), Vaswani et al. (2017), Brown et al. (2020), and Hoffmann et al. (2022).Source: §2 · p. 3

Sample-based evaluation of text generative models

• Text generation evaluation lacks an established, universally accepted sample-based metric, and prior proposed metrics showed vulnerabilities.The paper reports both sample likelihood according to a strong autoregressive model and human preferences as a middle ground.Source: §2 · p. 3

3.1 DIFFUSION MODELS FOR UNCONDITIONAL TEXT GENERATION

• A neural network estimates the clean data at each step to define learned Gaussian reverse transitions, and training minimizes a mean-squared-error objective.The simplified objective compares the clean data x0 with its estimates across diffusion steps.Source: §3.1 · p. 3
• SED embeds tokens in a fixed continuous space and applies diffusion there, rather than directly diffusing discrete token values.A forward discrete-to-continuous step maps tokens to Gaussian-distributed embeddings; a reverse readout maps the resulting continuous representation back to token probabilities.Source: §3.1 · p. 4
• A learned readout converts the continuous estimate into token logits, and a reconstruction cross-entropy loss trains that readout while the embedding matrix remains fixed.The paper says learning the embedding matrix was empirically unstable and could reduce unigram entropy.Source: §3.1 · p. 4
• Sampling runs 1,000 reverse steps and decodes each final continuous embedding through the readout, making it expensive relative to cached autoregressive sampling.Each forward pass processes all timesteps at once and is parallelizable; the authors identify reducing the number of sampling steps as a potential improvement.Source: §3.1 · p. 4
• Self-conditioning refines clean-data estimates by feeding the previous sampling-step estimate back into the denoising network.The estimate is concatenated with the current noisy input; training uses a first pass with self-conditioning set to zero and a second pass using a stop-gradient estimate.Source: §3.1 · p. 4

3.2 SPAN MASKING AND GUIDANCE FOR CONDITIONAL TEXT GENERATION

• SED creates varied conditioning and infilling tasks by randomly splitting sequences into fixed conditioning tokens and tokens subject to diffusion.Alternating spans are assigned to conditioning and infilling, with the mask flipped half the time; the single-span case is unconditional.Source: §3.2 · pp. 4–5
• Classifier-free guidance combines conditional and unconditional predictions to improve conditional generations.The method drops conditioning during training and extrapolates sampling toward the conditional prediction and away from the unconditional prediction.Source: §3.2 · p. 5

4.2 VALIDATION

• SED is trained on C4 with a non-causal transformer; validation combines 70B AR NLL, validation-target unigram entropy, and human judgments because automated metrics remain imperfect.Training uses a 32,000-token SentencePiece tokenizer and sequence lengths of 256 (128 for ablations); 10% padding tokens enable variable-size generation. Human judgments come from six colleagues evaluating 20 sample pairs per comparison.Source: §4.1 · p. 5, §4.2 · p. 6

4.3 RESULTS

• A single SED model supports unconditional generation and varied infilling, producing syntactically correct, coherent long sequences and conditioned spans with coherent transitions and diversity.The paper describes filling the middle or several spans as well as unconditional generation.Source: §4.3 · p. 6
• On suffix infilling, both SED sizes perform strongly against AR baselines, though humans prefer equal-size AR; SED-L receives 38% preference in infilling and 44% unconditionally.Table 4 reports the human-preference comparisons.Source: Figure 1 · p. 6, §4.3 · p. 7, Table 4 · p. 8
• Guidance improves sample quality and AR NLL, but scales above 4 yield repetition and lower entropy despite the NLL gain.The comparison uses guidance scales {1, 2, 4, 8} for SED and nucleus-sampling top-p values {1.00, 0.95, 0.90, 0.85} for AR.Source: Figure 1 · p. 6, §4.3 · p. 7

4.4 ABLATIONS

• Self-conditioning substantially lowers AR NLL, while pretrained embeddings further improve on random embeddings; together, they yield globally coherent, on-topic paragraphs with proper sentence structure.Random embeddings show topic-related word co-occurrences but can produce globally incoherent paragraphs and meaningless tokens.Source: §4.4 · p. 7
• Increasing pretrained embedding dimension can eventually degrade performance; low-dimensional diffusion more often passes through semantically related neighbors before becoming random.The authors hypothesize this destroys information more meaningfully and makes denoising easier.Source: §4.4 · p. 8
• A relatively high maximum span count is optimal and span masking improves unconditional NLL; scaling from SED-S to SED-L improves AR NLL from 4.20 to 3.68.The larger model also receives 63% human preference, compared with the smaller model.Source: §4.4 · p. 8

5 LIMITATIONS

• The authors say further tuning, larger models, and hyperparameter optimization are needed to better understand SED’s limits and compare it with state-of-the-art autoregressive models.They specifically note that the training regime would benefit from more hyperparameter optimization.Source: §5 · p. 9
• SED’s sampling is very inefficient, and reducing the required sampling steps is a priority for improvement.The paper notes that image diffusion methods have reduced sampling steps, but those advances remain largely unexplored for SED.Source: §5 · p. 9
• SED relies on a pretrained embedding space and a second model, and those embeddings may not be optimal for diffusion.The authors would ideally train the full model end-to-end, but report difficulty avoiding competition between diffusion and reconstruction losses.Source: §5 · p. 9
• Text-generation evaluation remains limited because NLL and unigram entropy can be gamed, NLL depends on an autoregressive evaluator, and clean infilling benchmarks are lacking.The authors also note that human evaluation confirmed their results, while arguing that a clean infilling benchmark would make them more convincing.Source: §5 · p. 9

6 CONCLUSION

• Span-masking tasks improve unconditional text generation in the reported ablation.Source: Table 8 · p. 10
• SED performs conditional and unconditional text generation and rivals autoregressive models while supporting flexible tasks such as infilling.The paper describes SED as a generally capable continuous diffusion model for text generation.Source: §6 · p. 10
• Promising directions include faster sampling, better embedding spaces for diffusion, and further investigation of infilling capabilities.Source: §6 · p. 10

A MODEL ARCHITECTURE

• Noised embeddings are projected independently to the transformer width, while a sinusoidal diffusion-step embedding is transformed and added to them.The step embedding passes through a dmodel × dmodel linear layer before addition.Source: A MODEL ARCHITECTURE · p. 12
• A linear output projection maps each transformer output back to the word-embedding size.Source: A MODEL ARCHITECTURE · p. 12
• With self-conditioning, the model concatenates the noisy input and its predicted clean version before the input projection; architecture hyperparameters are listed in Table 9.Source: A MODEL ARCHITECTURE · p. 12, Table 9 · p. 12

B FORWARD DIFFUSION PROCESS VISUALIZATION

• The visualization maps each noised token to its nearest embedding neighbor and colors it by its rank among the starting token’s 128 nearest neighbors.Figure 2 shows the first 16 neighbors; Figure 3 illustrates the forward process in dimension 896.Source: B FORWARD DIFFUSION PROCESS… · p. 12, Figure 2 · p. 13, Figure 3 · p. 13
• In lower dimensions, noised tokens’ nearest neighbors can become semantically related words, while high-dimensional diffusion jumps from the original token to an apparently random one.The authors hypothesize that the high-dimensional embedding space is mostly empty, while lower-dimensional diffusion destroys information more gradually.Source: B FORWARD DIFFUSION PROCESS… · p. 12

• Figure 4 visualizes forward diffusion for 300 steps using embeddings of dimension 32.Source: Figure 4 · p. 14
• Figure 5 shows the reverse diffusion process for SED-L with guidance scale 2.5.Source: Figure 5 · p. 14
• Table 10 presents samples from SED-L.Source: Table 10 · p. 15