Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?

1 Introduction

• Autoregressive models remain the default for text despite left-to-right, one-token-at-a-time generation and inability to revise earlier decisions; diffusion offers a path to alleviating these drawbacks.The paper focuses on continuous diffusion approaches, which avoid token factorization error and reuse techniques developed for continuous domains.Source: §1 · p. 1
• Unlike continuous data on a manifold, discrete data produce distinct density modes near data points as t → 0, where score errors and time discretization can trap samples between modes.The paper contrasts this with continuous-data models that recover the underlying distribution and data manifold.Source: §1 · pp. 1–2
• The paper frames late-stage sampling, when likelihood becomes multimodal and error sensitivity rises, as the critical phase for understanding Gaussian diffusion failures on discrete data.It uses this perspective to explain why modified solvers and self-conditioning can help.Source: §1 · p. 2
• In a critical interval, DDPM can enter low-likelihood regions and produce out-of-distribution inputs; the paper uses a Random Hierarchy Model to analyze this behavior.The authors explain why self-conditioning and q-sampling mitigate the problem.Source: §1 · p. 2
• The authors validate their findings across conditional and unconditional tasks in text, code, and proteins, and recommend switching from DDPM to q-sampling and tuning self-conditioning probability.They also report that combining the two heuristics improves generation quality on real data.Source: §1 · p. 2

2 Background

• Unconditional text generation samples a variable-length token sequence w from p(w), while conditional generation seeks text consistent with an observed variable y under p(w|y).Source: §2 · p. 2
• Gaussian diffusion progressively noises a continuous data sample toward a standard Gaussian, and trains a model to reconstruct clean data from noisy latents.Noisy latents can be sampled directly from x0 using q(xt|x0); the model is trained with a denoising objective.Source: §2 · pp. 2–3
• Continuous text diffusion encodes input sequences into latents, commonly with an embedding layer, and uses a linear decoder to recover tokens.The paper commonly trains embeddings jointly with the model, while pre-trained embeddings are also possible.Source: §2 · p. 3
• The DDPM solver reverses diffusion through a Markov chain with Gaussian transitions, starting from a standard Gaussian sample.Source: §2 · p. 3

3.1 Setup description

• The Random Hierarchy Model uses length-8 sequences over 16 tokens, of which 16,384 satisfy its generation rules; correctness is the rate of valid generated sequences.The finite data distribution is represented as a mixture of delta functions, allowing analytical computation of the noised density.Source: §3.1 · p. 3

3.2 Exploratory experiments

• Around t/T = 0.4, modes form and low-density regions emerge; incorrect trajectories can enter them, become out of distribution, and fail to denoise.Incorrect trajectories fall behind in correctness and density during the transition interval t/T ∈ [0.4, 0.5].Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Mitigating DDPM failures

• Branching into multiple noise realizations helps when activated early enough: at tact/T ≥ 0.4, correctness approaches 1 for sufficiently large candidate sets.At tact/T = 0.05, increasing candidate-set size gives no improvement because the sequence is nearly formed.Source: §3.3 · p. 5, Figure 3 · p. 5
• MBR improves correctness when applied within the transition interval; at tact/T = 0.4, correctness improves while diversity remains high.Its gain diminishes at larger activation timesteps as candidate samples become too diverse.Source: §3.3 · p. 5, Figure 3 · p. 5
• Switching from DDPM to q-sampling near the transition interval improves correctness over both pure DDPM and pure q-sampling, at the cost of reduced diversity.For previously incorrect predictions in the critical interval, q-sampling improves correctness and produces higher-density updates; it should not be used in early generation.Source: Figure 3 · p. 5, §3.3 · p. 6
• Self-conditioning improves correctness within the critical interval by anchoring predictions to previous predictions, but reduces diversity along with other mitigation methods.It reduces distances between consecutive predictions and can provide another input when the current latent is out of distribution.Source: §3.3 · pp. 6–7, Figure 5 · p. 7

Minimum Bayes-Risk

• MBR improves both Mauve and perplexity on ROCStories across all three diffusion models, but drastically slows generation.The authors therefore omit MBR from further experiments.Source: §4 · p. 8

Self-conditioning

• SC substantially improves generation quality across all three models, with the strongest gains for embedding-based models that fail without it.SC improves perplexity at some cost to diversity, yielding a better Mauve trade-off.Source: §4 · p. 8

Self-conditioning probability

• Tuning SC probability to p = 0.95 improves generation quality with only a 1.13× increase in training time over the p = 0.5 baseline.The authors propose increasing p closer to 1.Source: §4 · pp. 8–9

q-sampling

• Combining q-sampling with SC achieves the best overall performance on unconditional generation.Both techniques improve generation quality, and q-sampling provides a better perplexity–diversity trade-off.Source: §4 · p. 9
• Tuning q-sampling activation time within t_act ∈ [0.3, 0.5] improves quality with and without SC and requires no retraining.Unlike SC, q-sampling does not modify the training protocol or increase training time.Source: §4 · p. 9

Conditional generation

• On XSum, SC yields a much larger gain and the optimal q-sampling strategy activates from the beginning; on CoNaLa, SC gains are smaller and optimal activation is t_act/T ≈ 0.5.The authors hypothesize that both techniques help more when conditioning is weaker and the output can vary substantially.Source: §5 · p. 9, Figure 6 · p. 9

6 Conclusion

• DDPM trajectories tend to enter low-density regions during the critical, multimodal final phase; combining SC and q-sampling substantially improves quality on shallow-embedding models.The reported improvement is smaller for context-dependent embedding models.Source: §6 · p. 10

A Related work

• Earlier work examines noise schedules, SC, and q-sampling, while a one-hot-token analysis does not directly transfer to the embedding settings considered here.The paper frames its analysis as complementary to prior work on text diffusion failures.Source: A Related work · p. 17

B Random Hierarchy Model: setup description

• The RHM generates sequences through a hierarchy of levels, with equivalent expansions at each level; its specified setup uses sequence length n = 8.The setup uses L = 3, m = 4, s = 2, and V = 16.Source: B Random Hierarchy Model: setup… · p. 17

C Variance of latents updated with q-sampling and DDPM

• The variance analysis compares q-sampling and DDPM updates under the assumption that x_t is drawn from the marginal density p(x_t).It reports that both solvers have marginal variances below the forward process, with q-sampling lower still.Source: C Variance of latents updated with… · p. 18

C.1 Setup

• The variance comparison considers a finite-support data distribution and uses Bayes-optimal predictions for both reverse sampling procedures.The setup defines V_t as average posterior uncertainty.Source: §C.1 · p. 18

Variance comparison

• The DDPM update variance exceeds q-sampling’s by b^2 ᾱ_t V_t, while both solvers underestimate the true forward marginal variance.V_t denotes average posterior uncertainty.Source: §C.2 · pp. 18–20

When the solvers differ

• The variance gap vanishes when posterior uncertainty tends to zero or ᾱ_t tends to zero, and is largest at intermediate noise levels.The gap is largest when the posterior is genuinely uncertain but ᾱ_t remains appreciable.Source: §C.2 · p. 20

Why the solvers differ

• Q-sampling discards posterior uncertainty through fresh-noise resampling, whereas DDPM partially recovers it through coupling between its mean and x_t.The q-sampling variance is lower than DDPM’s.Source: §C.2 · p. 20

D Implementation details

• Models retain prior tokenization, architectures, and hyperparameters, with modifications for self-conditioning, code generation, and generation-step counts.Self-conditioning can be enabled or disabled at inference without retraining; the CodeFusion setup is reimplemented with GENIE, omitting pretraining to reduce compute; generation steps are reduced for several models.Source: D Implementation details · p. 20

E.1 Unconditional generation

• The datasets described are ROCStories (98 161 instances; 93 161 training and 5 000 validation), English Wikipedia (6.4M sequences; 10k validation), and OpenWebText (8M texts truncated to 512 tokens; 100k validation).Source: §E.1 · p. 21

E.2 Conditional generation

• Conditional tasks include summarization, question-pair paraphrase generation, German-to-English translation, and Python code generation.The datasets are XSum, QQP, IWSLT14, and CoNaLa, respectively.Source: §E.2 · p. 21

F Minimum Bayes-Risk examples

• The paper presents MBR-selection examples for the Random Hierarchy Model and ROCStories.Source: F Minimum Bayes-Risk examples · p. 21

F.1 Random Hierarchy Model

• At t_act/T = 0.3 or 0.5, MBR selects the correct RHM sequence, whereas performance depends on candidate diversity at other activation times.At 0.1 and 0.3, candidates are nearly identical; at 0.5 they include correct and incorrect sequences; at 0.7 they are too diverse for reliable averaging.Source: §F.1 · pp. 21–22, Table 4 · p. 22

F.2 ROCStories

• On ROCStories, MBR is most effective around t_act = 0.5, when candidates vary enough for useful selection but retain comparable content.At 0.3, candidates differ in only a few words; at 0.7, they diverge too much for reliable selection.Source: §F.2 · p. 22

• Self-conditioning consistently improves Mauve and reduces perplexity, while decreasing diversity; q-sampling activation produces a Mauve peak but earlier activation mostly degrades perplexity and diversity.The reported trends cover ROCStories, Wikipedia, and OpenWebText; Figures 7 and 8 show the activation-timestep comparisons.Source: G q-sampling ablation · p. 22, Figure 7 · p. 24, Figure 8 · p. 25
• Without SC, the peak occurs at tact/T = 0.3 for Diffusion-LM and approximately 0.3–0.4 for TEncDM Emb and Enc; with SC, preferred thresholds are 0.3 for Diffusion-LM and 0.5 for TEncDM Emb.With SC, q-sampling yields only marginal improvement over pure DDPM for TEncDM Enc regardless of tact; activating q-sampling at tact/T = 0.7 degrades performance in all settings.Source: G q-sampling ablation · pp. 22, 25

H Changing the number of steps

• With self-conditioning, increasing generation steps reduces perplexity and diversity, whether or not q-sampling is used.This is attributed to a growing self-conditioning mismatch as the number of steps increases.Source: H Changing the number of steps · p. 25, Figure 10 · p. 27
• Without self-conditioning, step count has little effect under DDPM, but more steps increasingly reduce perplexity and diversity as the q-sampling interval grows.Q-sampling lowers variance at each update, so more q-sampling updates leave the final latent closer to the distribution’s center and less committed to a text. Lower perplexity can raise Mauve for TEncDM Emb but lower it for TEncDM Enc.Source: H Changing the number of steps · p. 25

TEncDM Enc (With SC)

• For TEncDM models with self-conditioning, Figure 10 compares quality metrics across q-sampling activation times and generation-step counts on ROCStories and Wikipedia.The figure uses the baseline self-conditioning model with p = 0.5.Source: Figure 10 · p. 27

I Self-conditioning probability

• Increasing the self-conditioning training probability generally lowers perplexity, while diversity first rises and then falls; Mauve peaks near p = 0.9, but the best value varies by model and dataset.The authors recommend starting at p ≥ 0.8 and tuning upward.Source: I Self-conditioning probability · p. 28, Table 6 · p. 28

J Unsuccessful ideas

• Adding variance to x0 helped on the Random Hierarchy Model but did not improve real-data quality; training with learnable rather than frozen embeddings also produced equivalent quality.The authors suggest embeddings may converge early, limiting how much they vary during training.Source: J Unsuccessful ideas · pp. 28–29
• Resampling xt for the second self-conditioning prediction during training, with an input flag, did not improve generation quality.The experiment targeted the difference that xt stays fixed between the paired training predictions but updates between generation steps.Source: J Unsuccessful ideas · p. 29

K Additional results for unconditional generation

• Additional unconditional experiments use OpenWebText to test longer sequences and proteins to extend the analysis beyond text.Source: K Additional results for unconditional… · p. 29

K.1 OpenWebText results

• On OpenWebText sequences of length 512, self-conditioning gives a substantial performance boost, and q-sampling improves quality with or without self-conditioning at a diversity cost.Both techniques reduce perplexity while reducing diversity; the passage reports that self-conditioning is especially important on these longer sequences.Source: §K.1 · p. 29, Table 7 · p. 29

K.2 Protein generation results

• The protein experiments test whether text-side quality improvements from self-conditioning and q-sampling extend to another discrete domain, using the SwissProt subset of UniProt.The setup uses DiMA-35M trained on shallow ESM-2 150M embeddings and 100 sampling steps.Source: §K.2 · p. 29
• On SwissProt, both heuristics reduce pseud perplexity and FD-seq, but q-sampling decreases diversity while self-conditioning tends to increase it.Self-conditioning substantially improves FD-seq, whereas q-sampling yields only a marginal FD-seq gain; earlier q-sampling activation reduces pseudoperplexity and diversity, with optimal FD-seq at an intermediate activation time.Source: Table 8 · p. 29, §K.2 · p. 30, Figure 11 · p. 30

L Additional results for conditional generation

• For IWSLT-14, self-conditioning degrades performance and pure DDPM beats q-sampling at every activation time; for QQP, self-conditioning modestly improves BLEU while BLEU stays nearly constant across activation times.These outcomes differ qualitatively from XSum and CoNaLa.Source: L Additional results for conditional… · p. 30, Table 9 · p. 31, Figure 12 · p. 31
• The authors hypothesize that strong conditioning guides sampling away from low-density regions, and advise testing pure DDPM before adding self-conditioning, which adds cost and may hurt quality.They note that determining whether conditioning is strong enough may not be straightforward.Source: L Additional results for conditional… · pp. 30–31

M Image diffusion example

• On CIFAR-10, q-sampling degrades image quality and produces smoother samples; self-conditioning and MBR favor precision or typicality at the cost of diversity.The paper contrasts this with discrete domains, where avoiding low-density regions can help; on CIFAR-10 the same heuristics do not provide the same benefit.Source: M Image diffusion example · pp. 31–33, Figure 13 · p. 32, Table 11 · p. 32

N Image toy example

• The patch-based image task keeps outputs continuous while inducing discrete structure by composing image patches under a finite patch bank and transition rules.The transition-rule setup limits training to 1,000 images and uses |S| = 10 with r ∈ {4, 5}.Source: N Image toy example · p. 33
• Correctness measures valid outputs, while diversity measures unique correct outputs absent from training, distinguishing rule-following from novel valid combinations.Generated images are mapped to ordered patch identities before these measures are computed.Source: N Image toy example · p. 33
• In the patch-based tasks, q-sampling controls the correctness-diversity trade-off, with earlier activation reducing diversity; self-conditioning improves correctness without a comparable diversity drop.The q-sampling effect becomes pronounced around tact/T ≥ 0.3, and applying it atop a self-conditioned model again controls the trade-off.Source: N Image toy example · p. 34, Figure 14 · p. 34