PaperlayerView exact revision on arXiv

Source-linked AI summary

Remasking Discrete Diffusion Models with Inference-Time Scaling

Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, Volodymyr Kuleshov

arXiv:2503.00307v4cs.LGstat.ML

TL;DR

Masked discrete diffusion cannot revise tokens after they are generated, limiting iterative refinement. ReMDM adds remasking to sampling, and across language, image, and molecule tasks improves sample quality with more computation and enhances controlled generation.

  • Problem

    Masked discrete diffusion cannot revise tokens after unmasking, limiting its ability to iteratively refine generated outputs.

    Source: §1 · p. 1

  • Method

    ReMDM is a sampler derived from a custom remasking backward process that lets pretrained masked diffusion models remask generated tokens during sampling.

    Source: §1 · pp. 1–2

  • Results

    Across language, discretized images, and molecule string representations, ReMDM improves sample quality with more computation and enhances controlled generation and downstream performance.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Inference-time compute scaling is a consequential capability for masked diffusion, where that axis is more limited in existing models.

    Source: §7 · p. 9

    Generative perplexity can be misleading for ReMDM because schedules can lower it while sacrificing sentence entropy and diversity.

    Source: §5.1.1 · p. 6

Abstract

from arXiv · show

Part of the success of diffusion models stems from their ability to perform iterative refinement, i.e., repeatedly correcting outputs during generation. However, modern masked discrete diffusion lacks this capability: when a token is generated, it cannot be updated again, even when it introduces an error. Here, we address this limitation by introducing the remasking diffusion model (ReMDM) sampler, a method that can be applied to pretrained masked diffusion models in a principled way and that is derived from a discrete diffusion model with a custom remasking backward process. Most interestingly, ReMDM endows discrete diffusion with a form of inference-time compute scaling. By increasing the number of sampling steps, ReMDM generates natural language outputs that approach the quality of autoregressive models, whereas when the computation budget is limited, ReMDM better maintains quality. ReMDM also improves sample quality of masked diffusion models for discretized images, and in scientific domains such as molecule design, ReMDM facilitates diffusion guidance and pushes the Pareto frontier of controllability relative to classical masking and uniform noise diffusion. We provide the code along with a blog post on the project page: https://guanghanwang.com/remdm

1 Introduction

ReMDM lets masked discrete diffusion revise generated tokens by remasking them, and the paper presents it as a way to improve sample quality and controllable generation with additional inference-time computation.

Introduction

  • Masked discrete diffusion cannot revise an unmasked token, even when it introduces an error, limiting controllable generation, sampling speed, and sample quality.Source: §1 · p. 1
  • ReMDM adds remasking during generation to enable iterative refinement, lets users specify each step’s remasking probability, and gains performance from components such as nucleus sampling and remasking schedules.Source: §1 · pp. 1–2
  • ReMDM is ancestral sampling in a discrete diffusion model with a remasking backward process, whose objective is a rescaled MDLM objective [41], supporting its use with pretrained MDLMs.Source: §1 · pp. 1–2
  • The sampler can also be interpreted as a predictor–corrector method [53] [5] and is theoretically more general than prior masked-diffusion correctors [5] [13].Source: §1 · p. 2
  • With more inference steps, ReMDM improves language-model sample quality beyond prior diffusion models and nearly matches a same-architecture autoregressive model, while losing less quality at reduced steps.It also scales better than MaskGiT [7] on images and pushes the novelty–desired-property Pareto frontier in molecule design relative to masking or uniform-noise diffusion.Source: §1 · p. 2

2 Background

Discrete diffusion learns to reverse a fixed noising process; masked diffusion has a locked-token limitation because decoded tokens cannot be remasked or revised.

Discrete Diffusion Models

  • Discrete diffusion models learn to reverse a fixed noising process, extending diffusion to discrete signals such as one-hot vocabulary tokens [2] [41] [48].For finite-time processes, T denotes the number of steps; the noise schedule α_t decreases monotonically with t.Source: §2 · pp. 2–3

Masked Diffusion Language Models

  • Masked diffusion sets the limiting prior to a special mask token m, and once a token is unmasked, the reverse process keeps it unchanged.The authors call this the failure-to-remask property; they note it is shared with autoregressive models.Source: §2 · p. 3
  • Masked diffusion parameterizes pθ(z_s|z_t) by applying the forward posterior to neural-network predictions xθ(z_t) and trains with a variational objective [41].Source: §2 · p. 3

3 Remasking Diffusion Models

ReMDM extends masked diffusion with a posterior that can remask decoded tokens, preserves the marginals of classical masked diffusion (Sahoo et al. [41]), and supports reuse of pretrained MDLM weights.

3 Remasking Diffusion Models

  • ReMDM introduces a masked-diffusion model whose denoising posterior supports remasking decoded tokens during generation.The parameter σ_t controls remasking; setting it to zero recovers the MDLM posterior.Source: §3 · p. 3, §3.1 · p. 4

3.1 ReMDM Probability Decomposition

  • ReMDM preserves classical masked-diffusion marginals while enabling remasking, which makes its process non-Markovian, as in DDIM [51].The model retains the marginals of classical masked diffusion (e.g., Sahoo et al. [41]); conditioning transitions on x permits latents to move from masked state m back to x.Source: §3.1 · pp. 3–4

3.2 Negative Evidence Lower Bound

  • With σ_t = 0, ReMDM recovers the MDLM objective; its NELBO is looser, but pretrained MDLM weights can be reused and test perplexity is comparable.The paper reports weight reuse across different training and inference σ_t settings as performant in practice.Source: §3.2 · p. 4

4 ReMDM Samplers

ReMDM adds adjustable remasking to masked diffusion sampling, including ways to schedule or activate it and a formal connection to predictor-corrector samplers. Using σt = 0 during training also allows its sampler to work with a pretrained MDLM without retraining.

4 ReMDM Samplers

  • Using σt = 0 during training makes ReMDM compatible with a pretrained MDLM model and works without retraining it.The paper recommends using a different σt at inference, as one might use a different noise schedule during sampling.Source: §4 · p. 4

4.1 Design Strategies for the Remasking Schedule σt

  • ReMDM-cap limits the maximum remasking probability to ηcap, while ReMDM-rescale tempers σmax by a hyperparameter ηrescale.Both schedules provide alternatives for controlling the probability of remasking during sampling.Source: §4.1 · pp. 4–5
  • ReMDM sampling initializes masked tokens, iterates over timesteps, computes an approximate posterior, and samples the next state using a chosen remasking schedule.Algorithm 1 takes a pretrained denoising network, timestep count, noise schedule, and remasking schedule as inputs.Source: §4.1 · p. 5
  • ReMDM-conf assigns greater remasking probability to tokens with lower stored decoding confidence, while masked tokens are decoded using the MDLM approximate posterior.The confidence score is the token’s decoding probability when it was last unmasked.Source: §4.1 · p. 5

4.2 Design Strategies for ‘Turning On’ ReMDM

  • ReMDM-switch activates remasking after a chosen timestep, while ReMDM-loop first generates with MDLM, then remasks at constant noise to fix mistakes, and finally predicts remaining tokens with MDLM.The loop is intended to correct bad tokens while maintaining good ones, and can use any Section 4.1 remasking strategy in its remasking phase.Source: §4.2 · pp. 5–6

4.3 Comparison with Discrete Predictor-Corrector Samplers

  • Theorem 4.1 expresses ReMDM’s transition as an MDLM predictor followed by a corrector; FB [5] and DFM [13] correctors are special cases, while ReMDM supports constant-noise schedules unavailable to DFM.The FB corrector uses σt = αs−αt, and the DFM corrector can be converted using σt = βt(αs−αt).Source: §4.3 · p. 6

5 Experiments

ReMDM improves sampling quality and inference-time scaling across text and image generation, guided molecule generation, and diffusion language model benchmarks. Its results also show that remasking and confidence-based scheduling contribute to these gains, while generative perplexity alone can misrepresent text quality.

5.1.1 Text Generation

  • Generative perplexity can be tuned arbitrarily low while sacrificing sentence entropy and diversity, so the experiments use MAUVE as the primary text-quality metric.An extreme example achieves Gen PPL. 1.5 by generating repetitive text such as “love love love...”; MAUVE balances perplexity and diversity.Source: §5.1.1 · pp. 6–7
  • 15.62× MAUVE versus masked diffusion models and 2.23× versus MDLM with corrector samplers; ReMDM also better maintains quality under faster sampling.With more inference steps, masked diffusion models scale poorly and corrector samplers saturate.Source: §5.1.1 · p. 7, Table 1 · p. 7
  • Sequential ablations show that each proposed sampling improvement raises MAUVE over MDLM, with remasking providing the largest improvement.Figure 3 reports the MAUVE score for samples after each building block is ablated.Source: Figure 3 · p. 6, §5.1.1 · p. 7

5.1.2 Image Generation

  • At T = 64, ReMDM produces the highest-quality images among MaskGiT, MDLM, and ReMDM.The reported best settings use temperature 1.0 for MaskGiT and 0.8 for MDLM and ReMDM.Source: §5.1.2 · p. 8, Table 2 · p. 8
  • ReMDM-rescale slightly outperforms ReMDM-cap for image generation, and increasing the corresponding η parameters improves results for both schedules.The full parameter search is reported in Appendix E.3.Source: §5.1.2 · p. 8

5.2 ReMDM Improves Guidance

  • For both classifier-free and classifier-based guidance, ReMDM extends the novelty–property frontier beyond AR and diffusion baselines, with larger inference budgets improving results.The reported curves include only samples with at least 50 novel sequences; the best ReMDM strategy combines rescale and confidence schedules.Source: §5.2 · p. 8, Figure 4 · p. 8
  • For guided molecule generation, combining ReMDM-rescale with the confidence-based scheduler improves results.The parameter tuning identifies this combination as beneficial.Source: §5.2 · p. 9

5.3 ReMDM Improves Benchmark Performance

  • LLaDA with ReMDM performs best on Countdown and TruthfulQA and significantly exceeds the original LLaDA across repeated-seed experiments.Results are reported as means with 95% confidence intervals.Source: §5.3 · p. 9

6 Discussion and Related Work

The discussion reports that LLaDA with ReMDM performs best on downstream tasks and situates ReMDM among discrete predictor-corrector samplers. It distinguishes plug-and-play correctors from alternatives that require additional training.

Discussion and Related Work

  • LLaDA with ReMDM performs best on downstream tasks and consistently exceeds original LLaDA with statistical significance.Table 3 identifies the best value for each metric in bold, but the supplied passage gives no metric values.Source: Table 3 · p. 9
  • CTMC theory provides a framework for reverse-process error-correcting samplers, extending the continuous-data predictor-corrector formulation [53] to discrete domains.FB and DFM correctors are discussed alongside ReMDM.Source: §6 · p. 9
  • The Stein operator [67] and DPC are other correctors; DPC improves MaskGiT sample quality, and both require additional training unlike FB, DFM, and ReMDM.Source: §6 · p. 9

7 Conclusion … A.7 Proof of Proposition 4.3

ReMDM enables pretrained absorbing-state discrete diffusion models to remask predicted tokens, improving generation quality and control while adding an inference-time compute-scaling axis.

7 Conclusion

  • ReMDM enables remasking of predicted tokens in pretrained absorbing-state discrete diffusion models, improving sample quality, controlled generation, and downstream performance while adding an inference-time compute-scaling axis.The conclusion reports improvements for both unconditional and conditional generation.Source: §7 · p. 9

A.1 Proof of Theorem 3.1

  • Induction establishes that the absorbing-state marginal is q(z_s | x) = Cat(z_s; α_s x + (1 − α_s)m).The proof uses that z_t can only be x or the mask token m.Source: §A.1 · p. 16

A.2 Deriving the non-Markovian forward step qσ(zt | zs, x)

  • Bayes’ rule and the cases z_s = m and z_s ≠ m yield the non-Markovian forward step qσ(z_t | z_s, x).The derivation substitutes the marginals from Theorem 3.1 and combines the two cases.Source: §A.2 · pp. 16–17

A.3 Deriving Bounds for σt

  • Requiring posterior coefficients to lie in [0, 1] gives the σ_t bounds, with the monotonically decreasing noise schedule making the lower bound nonpositive.Source: §A.3 · p. 17

A.4 Deriving the ReMDM NELBO

  • The ReMDM NELBO diffusion-loss derivation recovers the diffusion loss term in equation (9).In practice, setting pθ(z1) = q(z1 | x) = q(z1) = π makes the prior loss zero, and the reconstruction loss matches equation (9).Source: §A.4 · pp. 17–18

A.5 Proof of Theorem 4.1

  • The predictor-corrector construction in Theorem 4.1 yields the ReMDM posterior in equation (6).Conditioned on the predictor sample, the corrector sample is independent of z_t.Source: §A.5 · pp. 18–19

A.6 Proof of Proposition 4.2

  • The forward-backward corrector sampler [5] combines forward and backward rate matrices, and its discretization is a special case of ReMDM with σ_t = α_s − α_t.The comparison rewrites the ReMDM step at time t to align the marginal being held fixed.Source: §A.6 · pp. 19–20

A.7 Proof of Proposition 4.3

  • The DFM corrector sampler [13] weights forward and backward sampling velocities using a user-specified schedule β_t, yielding the ReMDM posterior with σ_t = β_t(α_s − α_t).Source: §A.7 · pp. 20–21

A.8 Proof of Proposition 4.4 · B Comparison to DDIM Non-Markovian Processes

The proof shows that ReMDM strictly generalizes the DFM corrector, which can degenerate into copying existing tokens when the noise schedule is constant over an interval. The comparison with DDIM [51] adapts its discrete process to absorbing-state diffusion and establishes equivalence under a reparameterization.

A.8 Proof of Proposition 4.4

  • The proof states that the ReMDM posterior defined in equation (54) is a valid probability distribution.Source: §A.8 · p. 21
  • The DFM corrector velocity is a weighted combination of the forward and backward velocities.Source: §A.8 · p. 21
  • When α_t remains constant over an interval, the DFM corrector becomes a degenerate sampler that copies the existing token without changes.The proof states that both forward and backward velocities vanish in this interval.Source: §A.8 · p. 22
  • ReMDM strictly generalizes the DFM corrector, whose behavior is a special case of the ReMDM sampler.The proof establishes that the inclusion is strict.Source: §A.8 · p. 22

B Comparison to DDIM Non-Markovian Processes

  • Replacing DDIM’s uniform categorical limiting distribution with the absorbing-state distribution m adapts its discrete process to absorbing-state diffusion.DDIM [51] originally assumes a uniform categorical limiting distribution for discrete data.Source: B Comparison to DDIM Non-Markovian… · p. 22
  • ReMDM is equivalent to an adapted DDIM [51] absorbing-state process under a reparameterization, for both masked and non-masked cases.The equivalence is derived by comparing the posteriors for z_t ≠ m and z_t = m.Source: B Comparison to DDIM Non-Markovian… · p. 22
  • The ReMDM formulation is presented as easier to analyze than the reparameterized DDIM formulation, including for deriving bounds on σ_t and exploring design choices.The design choices referenced are those defined in Section 4.Source: B Comparison to DDIM Non-Markovian… · p. 22

C ReMDM Sampler Algorithms

ReMDM-switch alternates between a remasking phase and a phase with remasking disabled, then computes approximate posteriors for the sequence. The algorithms may also use strategies for setting σt described in Section 4.1.

C ReMDM Sampler Algorithms

  • ReMDM-switch takes a pretrained denoiser, sequence length, timestep count, noise schedule, remasking controls, confidence option, and switch time as inputs, then initializes the sequence.Source: C ReMDM Sampler Algorithms · p. 23
  • Before the switch time, the remasking value is ηrescale · min{ηcap, (1 − αs)/αt}; after it, the algorithm sets remasking to zero.Source: C ReMDM Sampler Algorithms · p. 23
  • For each sequence position, the algorithm computes an approximate posterior and samples from pθ; ReMDM-switch and ReMDM-loop can optionally use strategies for setting σt.Source: C ReMDM Sampler Algorithms · pp. 22–23

D Additional Experimental Details … E Additional Experimental Results

The supplied passages document the experimental setups and evaluation procedures for OpenWebText, ImageNet, QM9, and LLaDA; they do not provide the additional-results findings.

D.1 OpenWebText

  • OpenWebText reuses 169M-parameter AR, SEDD, and MDLM Transformers with 12 layers, 12 attention heads, and 768 hidden dimensions, from checkpoints released by [41].The architecture augments the diffusion transformer [35] with rotary embeddings; its full details are attributed to Sahoo et al. [41].Source: §D.1 · p. 23
  • The evaluation reports MAUVE scores from 5,000 samples per model/sampler, generative perplexity with GPT-2 Large as the external model, and mean sequence entropy across 5,000 sequences; GPT-2 Large is the MAUVE embedding model.Entropy is computed on token counts before tokenizer decoding, and the MAUVE scaling hyperparameter is 5.Source: §D.1 · pp. 23–24

D.2 ImageNet

  • ImageNet comparisons define MaskGiT as confidence-based decoding [7], MDLM as the zero-mask parameterization with posterior (2), and ReMDM as the same denoiser with posterior (6).ReMDM separately tests max-capped, rescaled, and confidence-based schedules, with softmax temperatures τ ∈ {0.6, 0.8, 1.0}.Source: §D.2 · p. 25

D.3 QM9

  • QM9 guidance uses the full, un-approximated D-CBG implementation rather than Schiff et al.’s [46] first-order approximation, with DiT-based AR, MDLM, and UDLM models.Sequences have padded length 32 and vocabulary size 40; guidance labels mark molecules at or above the 90th-percentile property value as positive.Source: §D.3 · p. 25
  • QM9 ReMDM grid searches rescale strengths, optional confidence scheduling, switch or loop strategies, and sampling settings for both QED and ring count with D-CFG and D-CBG.Evaluation generates 1,024 sequences per model/guidance mechanism, removes invalid and duplicate molecules, and reports novelty plus mean QED or ring count.Source: §D.3 · pp. 25–26

D.4 LLaDA

  • ReMDM-loop adapts pretrained LLaDA sampling by remasking the lowest-confidence tokens for 32 steps, while DFM instead remasks one and unmasks two tokens per step.For maximum mask length 32, LLaDA first generates 28 tokens; the loop remasks 1 token for Countdown or 4 for TruthfulQA, then LLaDA fills remaining gaps. Random sampling is enabled for error bars.Source: §D.4 · p. 26
  • LLaDA evaluation uses 4-shot Countdown and 6-shot TruthfulQA, reporting mean pass@1 or the correct-minus-incorrect ROUGE difference with 95% confidence intervals.Countdown uses 256 questions with 10 trials each; TruthfulQA uses 817 questions, and both experiments are repeated across 20 random seeds.Source: §D.4 · p. 26

E.1 Training with MDLM v.s. ReMDM NELBO … E.4.1 Drug-likeness Property Maximization

Across these appendix studies, ReMDM training objectives yield similar QM9 validation perplexities, while sampling schedules and hyperparameters affect text-generation quality and ReMDM improves the QED novelty-property maximization frontier relative to MDLM. ImageNet results also show different preferred softmax temperatures for MDLM/ReMDM and MaskGiT.

E.1 Training with MDLM v.s. ReMDM NELBO

  • On QM9, models trained with the MDLM and ReMDM NELBO objectives have fairly consistent validation perplexities.The ReMDM objective uses σt = min{ηcap, (1 − αs)/αt}; the table compares continuous-time MDLM and discrete-time objective settings.Source: §E.1 · p. 26, Table 4 · p. 26

E.2.1 ReMDM σt v.s. DFM σt

  • DFM remasking schedules spike initially and then have a long tail, using βt = At^0.25(1 − t)^0.25 with time-step widths adjusted accordingly.This is the best-performing DFM schedule reported in Gat et al. [13].Source: §E.2.1 · p. 27, Figure 5 · p. 27

E.2.2 Hacking Generative Perplexity

  • Increasing η can sharply reduce Gen PPL. at the cost of entropy and diversity, whereas MAUVE cannot be hacked this way.The comparison uses ReMDM-cap on OWT with T = 1024 and L = 1024, generating 5000 samples per experiment.Source: §E.2.2 · p. 27, Table 5 · p. 27

E.2.3 Comparing Different ReMDM Strtegies

  • For inference-time scaling (T ≥ L), ReMDM-loop performs best; for faster sampling (T < L), ReMDM-cap performs better, while ReMDM-rescale underperforms both.These are OpenWebText unconditional-generation comparisons across ReMDM schedules.Source: Table 6 · p. 28

E.2.4 Tuning ηcap / ηrescale

  • For ReMDM-cap, increasing ηcap lowers MAUVE at T = 4096 but generally raises it at T = 1024; for faster sampling, ηcap increases MAUVE except at 0.045.The study attributes the opposing trends to the perplexity–entropy trade-off and reports ηcap = 0.008 for inference-time scaling and η = 0.040 for faster sampling. For cap-plus-loop, it uses ηcap = 0.020 (T ≥ L) and 0.055 (T < L); for rescale, ηrescale = 0.015 (inference-time scaling) and 0.045 (faster sampling).Source: §E.2.4 · p. 28, Figure 6 · p. 29, Figure 7 · p. 29

E.3 ImageNet

  • MDLM and ReMDM benefit from τ = 0.8, whereas MaskGiT performs best at τ = 1; reduced temperature trades diversity against FID and IS.FID determines the reported “best” setting for each sampler; IS penalizes reduced diversity less than FID [16].Source: §E.3 · p. 28, Table 7 · p. 30

E.4 QM9

  • The QM9 hyperparameter search covers ring-count and QED properties with D-CFG and D-CBG guidance.Figures 10–25 report the search results across both properties and guidance mechanisms.Source: §E.4 · p. 30

E.4.1 Drug-likeness Property Maximization

  • ReMDM improves the novelty-property maximization frontier over MDLM for QED, though less than for ring-count maximization.The reported settings combine ReMDM-rescale with confidence-based scheduling and switching: ηrescale = 0.1 and tswitch = 0.1 for D-CFG, and ηrescale = 1.0 and tswitch = 0.5 for D-CBG.Source: §E.4.1 · p. 30

E.4.2 Tuning ηrescale … E.4.5 Comparing to FB and DFM Correctors

ReMDM improves the novelty–property trade-off over FB and DFM in conditional ring-count generation, while tuning results vary by property, confidence schedule, and guidance mechanism.

E.4.2 Tuning ηrescale

  • Figure 9 shows ReMDM extending the novelty–property maximization frontier for QM9 QED generation across inference compute T and guidance strength γ.The figure compares D-CFG with D-CBG and FUDGE for autoregressive models.Source: Figure 9 · p. 31
  • Larger ηrescale generally pushes ring-count Pareto curves toward more novelty and higher ring counts, with results less sensitive to ηrescale at values ≥0.5.The trend has exceptions without the confidence-based schedule, where ReMDM performs comparatively worse than with confidence.Source: §E.4.2 · pp. 30–31
  • For QED maximization, no ηrescale clearly dominates with confidence-based schedules, while without them results trend better at ηrescale = 0.1.Source: §E.4.2 · p. 31

E.4.3 Ablating Use of Confidence-based Schedule

  • Confidence-based schedules improve QED maximization and also help ring-count maximization, though the ring-count trend is less pronounced.For ring counts, confidence may temper large remasking probabilities for relatively safe tokens, especially at larger ηrescale.Source: §E.4.3 · p. 31

E.4.4 Tuning tswitch / Loop Parameters

  • With D-CFG, both QED and ring-count maximization generally benefit from activating ReMDM later, using smaller tswitch or ton.Source: §E.4.4 · p. 31
  • With D-CBG, later ReMDM activation benefits QED maximization, but ring-count maximization instead improves with earlier activation.The ring-count exception uses larger tswitch or ton.Source: §E.4.4 · p. 31

E.4.5 Comparing to FB and DFM Correctors

  • ReMDM better trades off novel-sample generation and ring-count maximization than FB [5] and DFM [13] under conditional generation with D-CFG (γ = 1).The comparison is reported for ring-count maximization.Source: §E.4.5 · p. 31

F Generated Samples … G Assets

The examples show ReMDM producing fluent, topic-focused English, while also exposing a failure mode in which repetitive, meaningless text achieves very low generative perplexity. The appendices include few-shot task examples, asset-license inventories, and hyperparameter-tuning visuals.

F.1 Generative Perplexity Hacking Example

  • Generative perplexity can be misleading: ReMDM-cap on OWT produces repetitive tokens without real semantics despite a generative perplexity of 1.5.The example uses ηcap = 0.2 and T = 1024; Figure 26 also reports entropy 1.04.Source: §F.1 · p. 31, Figure 26 · p. 48

G Assets

  • Supplementary visuals document ring-count hyperparameter tuning for ReMDM-rescaled schedule variants, including D-CFG and D-CBG settings.The figures vary loop or switch schedules and whether confidence-based schedules are used.Source: Figure 10 · p. 32, Figure 11 · p. 33, Figure 12 · p. 34

F.2 OpenWebText

  • ReMDM generates fluent English with a clear topic, unlike MDLM’s fragmented, often ungrammatical samples and MDLM+DFM’s unfluent internal logic.These comparisons use samples generated with T = 4096 diffusion steps.Source: §F.2 · p. 32, Figure 27 · p. 49, Figure 28 · p. 50

F.3 LLaDA

G Assets

Related papers

Loading 2503.00307v4…