Conditional [MASK] Discrete Diffusion Language Model

1 Introduction

• Autoregressive models face diversity and controllability challenges linked to sequential generation, which prevents revising earlier steps.The cited examples include difficulty correcting mathematical reasoning errors and integrating external knowledge.Source: §1 · p. 1
• CMLMs offer contextual understanding but lack an effective generation mechanism, while DDLMs can degenerate on conditional tasks despite iterative refinement.Source: §1 · p. 1
• A conditional Markov Random Field bridges one-step CMLM prediction and iterative diffusion, supporting stepwise generation and stable conditional text with reduced semantic drift.The overview presents the combination as targeting a better diversity-quality tradeoff and fine-grained controllability.Source: Figure 1 · p. 1, §1 · p. 2
• Entropy-Adaptive Gibbs Sampling updates the most uncertain tokens first at each denoising step.The authors describe this strategy as supporting qualified generation.Source: §1 · p. 2
• Entropy-based Noise Scheduling progressively masks tokens in ascending entropy order so the model learns a structured denoising process.Source: §1 · p. 2
• Across conditional generation tasks, Diffusion-EAGS improves over baselines and achieves the best quality-diversity tradeoff, with coherent, controlled keyword-based stories.The authors report that the model balances fluency and variability more effectively than existing models.Source: §1 · p. 2

2 Related Works

• Recent diffusion methods for NLP report promising performance across varied tasks, building on image-generation advances and diffusion's potential for controllability and diversity.Source: §2 · p. 2
• Continuous diffusion models such as Diffusion-LM, DiffuSeq, and LD4LG show promise, while Bansal et al. (2022) argue stochastic randomness need not govern these operations.Source: §2 · p. 2
• Discrete methods include restoration-generation and pretrained-language-model integration; later work improves score-based models, open-ended generation, infilling, and scalability.The passage names D3PM, DiffusionBERT, SEDD, and subsequent extensions and applications.Source: §2 · p. 2

3 MLM & DDLM : D-cMRF

• D-cMRF integrates discrete diffusion into MLM generation, using selective high-uncertainty updates to support stable, coherent generation.The framework combines MLM contextual representations with iterative denoising.Source: §3 · p. 2

3.1 MLM as cMRF

• The MLM is reinterpreted as a conditional MRF whose sequence energy uses MLM-based embeddings and whose token potential is conditioned on the other tokens and Y.The MLM logits provide the token-position potential; the partition function ensures proper normalization.Source: §3.1 · p. 3

3.2 DDLM with Entropy-based Denoising

• At each diffusion step, the method updates positions with the highest entropy using a dynamically adjusted threshold, sampling their replacements from the conditional sequence distribution.The authors identify sampling and energy updates as a challenge because the energy space is not guaranteed to remain stable.Source: §3.2 · p. 3

3.3 D-cMRF

• The paper argues that combining DDLM with cMRF guarantees energy reduction during generation and stable sequence reconstruction.The diffusion process integrates the conditional sequence distribution as a denoising function.Source: §3.3 · p. 3, §3.2 · p. 3
• Across diffusion steps, entropy-based selection yields a net decrease in sequence energy toward an approximate low-energy state despite possible stochastic fluctuations.The sequence energy is lower-bounded, and finite sequence length supports the stated convergence argument.Source: §3.3 · p. 4

3.3.1 D-cMRF Guarantees

• The stated guarantees are progressive energy reduction and stable convergence to a structured, low-entropy state; the paper also reports that removing EAGS significantly drops performance.The ablation claim is reported without numerical values in the supplied passage.Source: §3.3.1 · p. 4

4 Diffusion-EAGS

• Diffusion-EAGS pairs Entropy-based Noise Scheduling during training with Entropy-Adaptive Gibbs Sampling during generation for balanced text quality and diversity.ENS selectively masks tokens based on certainty; EAGS refines a fully masked, conditioned sequence by updating high-uncertainty tokens.Source: §4 · p. 4

4.1 Inference Process: Entropy-Adaptive Gibbs Sampling

• The selected positions M_t also address the MLM's difficulty in choosing which tokens to denoise next during DDLM sampling.The paper names this strategy Entropy-Adaptive Gibbs Sampling.Source: §4.1 · pp. 4–5
• EAGS ranks tokens by descending entropy and prioritizes the least informative positions, using established syntactic context to form more structured sequences.The procedure computes each variable's entropy before determining denoising positions.Source: §4.1 · p. 5

2. Variable Selection: Obtain Mt for sampling

• At each iteration, sample selected tokens from their conditional distributions, update affected distributions and entropy, and repeat until the final timestep.The selected positions are in M_t.Source: §2 · p. 5

4.2 Training Process: Entropy-based Noise Scheduling

• ENS masks the lowest-entropy positions first, selecting L/T positions per step to make the training forward process mirror generation.The entropy is computed using a pretrained MLM with the input sequence [Y | X(t)].Source: §4.2 · p. 5

4.3 Diffusion Loss with Cross Entropy

• The method omits PLM parameterization because it empirically restricts response diversity, and uses cross-entropy instead of mean squared error in the diffusion loss.The passage contrasts this choice with a PLM parameterization that preserves semantic embedding spaces during training.Source: §4.3 · p. 5

5.1 Tasks & Details

• Experiments use open-ended RocStories and strongly constrained ParaDetox, testing 20 conditions with five outputs each and selecting five samples from 20 by perplexity.The 100 test outputs are not used for training; generation uses five steps.Source: §5.1 · p. 5

5.2 Baselines

• Comparisons cover autoregressive, conditional masked, continuous diffusion, and discrete diffusion models, with RoBERTa-base as the stated model-size reference.The authors state that investigating architecture capabilities, rather than scalability or bypassing architectural limitations, is the primary goal.Source: §5.2 · pp. 5–6

5.3 Metrics

• Quality is assessed with PPL, MAUVE, SOME, MOS, and LLM-c, while diversity uses n-gram and embedding Vendi Scores and human MOS.PPL is based on GPT-2 Large and GPT-2 XL; MAUVE assesses style consistency with training data.Source: §5.3 · p. 6

• Diffusion-EAGS achieves lower PPL and strong MOS while handling conditional constraints, balancing fluency and text quality.The cited comparison describes high MAUVE and MOS scores and contrasts the model with systems that can struggle with semantic drift or high PPL.Source: §6 · p. 6
• On open-ended RocStories, Diffusion-EAGS maintains competitive scores compared with GPT-2, including under minimal constraints.Source: §6 · p. 7
• Diffusion-EAGS surpasses other baselines and larger LLMs on both reported Vendi diversity metrics, with human ratings also indicating more varied and engaging outputs.The metrics are n-gram and embedding-based Vendi Scores, alongside MOS for diversity.Source: Table 3 · p. 7, §6 · p. 7
• Overall, the reported results combine low perplexity and high human-evaluation scores with richly varied text across conditional generation tasks.Source: §6 · pp. 6–7

7.1 Quality-Diversity Tradeoff

• Diffusion-EAGS achieves the best quality–diversity tradeoff, outperforming prior approaches in diversity without compromising fluency.Figure 3 plots 1/PPL against VS_emb; the purple-star variants are reported to have the best overall tradeoff.Source: §7.1 · p. 7, Figure 3 · p. 7

7.2 Keyword Based Generation

• Generated sequences integrate specified keywords with context-specific semantics in the keyword-based generation examples.The examples place keywords in masked positions within supplied contexts.Source: §7.2 · pp. 7–8, Table 4 · p. 8

7.3 Ablation Study

• Removing EAGS causes a severe quality decline, while removing Gibbs sampling increases PPL and reduces diversity.The reported EAGS ablation resembles degenerate traditional CMLM outputs; the no-Gibbs condition removes the diffusion generation process.Source: §7.3 · p. 8, Table 5 · p. 8
• On Paradetox, PPL was 1193 for lowest-entropy selection, 183 for random selection, and 112 for highest-entropy selection.The authors adopted highest-entropy selection after heuristic evaluation aligned quality with these PPL values.Source: §7.3 · p. 8
• EAGS contributes to a gradual entropy decrease during generation, which the authors say enables stable sentence generation.The entropy-flow analysis is reported in Appendix D.Source: §7.3 · p. 8

8 Conclusions & Discussions

• Diffusion-EAGS integrates masked language models with diffusion models for conditional generation, with improved quality, diversity, applicability, and token-level control.The paper identifies exploring other pretrained language models as a future direction.Source: §8 · pp. 8–9
• A toy T5 experiment on Paradetox showed no significant improvement over GPT-2 fine-tuning.The authors hypothesize that T5 generation is strongly influenced by its initial decoder tokens and suggest further work on integrating encoder–decoder models with diffusion.Source: §8 · pp. 8–9

Limitations

• The method is currently focused on text generation; classification applicability and integration of autoregressive models remain unexplored, and pretrained bias can propagate.The authors note data-balancing strategies as a way to address bias and identify these issues as relevant to deployment.Source: Limitations · p. 9

A Necessity of Energy Update in cMRF Generation

• RocStories conditions are associated with increased log-potential values, supporting energy updates when conditioning changes.The paper notes that different conditions can correspond to different random fields; Figure 4 depicts a logarithmic-scale shift in potential values.Source: A Necessity of Energy Update in cMRF… · p. 13, Figure 4 · p. 13

B.2 Results of Experiment and Implications for Conditional Generation

• With BERT-large-cased, average log potential was 156.6150 for standard MLM and 175.5332 with RocStories guidance; fine-tuning on RocStories yielded 3.7551 on an exponential scale.The authors interpret these results as showing dataset structure’s influence on MLM potential and narrative coherence.Source: §B.2 · p. 13

1. Initial BERT Refinement Without a Noise Matrix

• Initial BERT refinement without a noise matrix failed to denoise masks, while a 0.15-masking-ratio BERT matrix biased outputs toward a repeated token.The reported examples include sequences with remaining masks and repeated “wwii” tokens.Source: §1 · p. 13, §2 · p. 13

5. Vocab-Wise Entropy Estimation

• Time-reversal scheduling yielded NA results under strict Paradetox conditioning, and word-frequency scheduling struggled to construct coherent sentences.Vocab-wise entropy estimation improved examples, but the MLM could remask generated tokens and replicate tokens during final filling.Source: §3 · p. 13, §4 · p. 14, §5 · p. 14

7. Entropy Selection Criteria

• For Paradetox, uniform, reverse-order EAGS, and EAGS produced PPLs of 182.976, 1193.229, and 112.190, respectively.The authors select EAGS, assuming a denoised mask remains fixed and preserving sampled tokens across subsequent diffusion steps.Source: §7 · p. 14, §6 · p. 14

D Entropy Flow

• Diffusion-EAGS shows a gradual generation-time entropy decrease, while Gibbs sampling without entropy tracking did not converge within 20 steps.The paper associates the gradual decrease with successful DDLM training and superior text quality; sampling instability is associated with lower average quality. Figure 5 illustrates entropy trends.Source: D Entropy Flow · p. 14, Figure 5 · p. 14

F.1 Fine-Grained Conditional Generation

• The paper distinguishes context continuity, required keywords, and semantic or formatting transformations as three levels of conditional constraints.It aims for a diffusion framework applicable across a wide range of conditional generation tasks.Source: §F.1 · p. 15

F.2 Dataset Explanations

• The evaluated datasets cover story continuation, deontology, paraphrase, question generation, dialogue summarization, detoxification, and English–German translation.The tasks vary in their conditioning needs, including context, specified content, and transformations.Source: §F.2 · p. 15

F.3 Experimental Details

• Evaluation uses perplexity, grammar and plausibility measures, MAUVE, and human ratings for quality, alongside Self-BLEU, distinct-n, and embedding Vendi Score for diversity.The reported setup also describes model, sequence-length, sampling, and annotation procedures.Source: §F.3 · pp. 15–16

G.1 Fine-Grained Comparison

• Across reported comparisons, Diffusion-EAGS maintains strong text quality alongside diversity; it has the highest MAUVE on ParaDetox and exceeds GPT models on VS(ngram) and VS(emb).The authors also report that the model produces more diverse sequences than Dinoiser and uses only a few generation steps; Tables 8 and 9 provide comparison results.Source: §G.1 · pp. 16–18, Table 8 · p. 17, Table 9 · p. 17

Paradetox w/ LLM-t on application models

• GPT-4 pairwise judgments favored Diffusion-EAGS over baselines despite a reported prompt position bias against it, and the model outperformed baselines on DialogueSum ROUGE.The DialogueSum result is presented in Table 10; the pairwise evaluation is presented in Table 13.Source: Paradetox w/ LLM-t on application models · p. 18, Table 10 · p. 18, Table 13 · p. 18

G.3 Machine Translation : Bilinguality & Low Resource Settings

• Translation results report that diffusion and non-autoregressive transformer baselines struggled to produce correct outputs, while Diffusion-EAGS showed promising results.The comparison is reported for English–German translation in Table 14.Source: §G.3 · p. 19, Table 14 · p. 19

G.4 Diversity Analysis

• Other diffusion models often rely on deletion or word-level paraphrase, whereas Diffusion-EAGS produces varied sentence forms and improves diversity MOS.The discussion also identifies semantic leaps or weak structural constraints as issues in conditional generation.Source: §G.4 · p. 19
• Figure 6 shows Diffusion-EAGS diversity rising more steeply, while GPT-series embedding VS saturates below 13.Fine-tuned GPT-2 also struggles to terminate sentences with <eos> under strongly constrained conditional generation.Source: §G.4 · pp. 19–20, Figure 6 · p. 20

G.5 Keyword Generation Results

• Table 4 examples show Diffusion-EAGS generating coherent sequences from the given context.Source: §G.5 · p. 20

G.6 Different PLM : BERT & T5

• BERT and RoBERTa show similar trends, while T5 behaves comparably to fine-tuned GPT-2; the authors suggest alternative methods may be needed for changed architectures.They conjecture T5’s autoregressive decoder training contributes to its relatively low diversity.Source: §G.6 · p. 20

K.1 Datasets & Observations

• On translation, neural metrics can treat outputs containing other languages as semantically coherent despite lower BLEU, and a gap between translation models and DDLM remains.The experiments use an 18k human-curated en↔de dataset and a pretrained NLLB model for non-autoregressive language control.Source: §K.1 · p. 22

K.2 Comparison Between Easy-First and Our Proposed Method

• Unlike Easy-First’s predictions based on previously predicted easier positions, the proposed method denoises masked states along the forward noising trajectory and supports PLM integration.Source: §K.2 · p. 22

K.4.1 NAR Transformer, DisCo

• DisCo performs poorly on low-resource translation; the authors associate the drop with the much smaller dataset than WMT14 EN-DE’s 4.5M pairs.Using a pretrained mBART dictionary gives a slight benefit but still trails other models.Source: §K.4.1 · p. 23

K.4.2 Diffuseq-v2

• In low-resource translation, Diffuseq-v2 has much higher token-prediction entropy and nearest-token distance ratios, suggesting simple rounding may be insufficient.Source: §K.4.2 · p. 23