Simplex Diffusion Models

• Categorical diffusion discards denoiser uncertainty at each update; DFM (Stark et al., 2024) relies on ODE/SDE integration and scales poorly to large vocabularies, while Simplax (Sakurai et al., 2026) samples categorically.Continuous relaxations can also destroy clean-category identity abruptly, with the problem worsening as encoding dimension grows.Source: §1 · pp. 1–2
• At 512 NFE, SDMs without SC reach 49.0% at T=0.1 vs. 45.8% for MDM + SC, and 45.8% at T=1 vs. 36.8% for UDM + PC.The T=0.1 comparison is on TinyGSM; the reported baselines use self-conditioning or predictor-corrector sampling.Source: Figure 1 · p. 2, §1 · p. 2
• SDMs use a Dirichlet forward process with separately parameterized mean and concentration, enabling closed-form reverse transitions and a DDIM-style sampler with tunable stochasticity.The churn parameter controls sampling stochasticity, and the model is trained with cross-entropy.Source: §1 · p. 2
• Distilled SDMs solve 32.1% of GSM8K problems in 8 steps, versus 21.4% for distilled Discrete Diffusion in 128 steps; on Sudoku, SDMs with SC reach 99.1% vs. 99.3%.On OpenWebText, SDMs are competitive with uniform and masked diffusion on the GenPPL/entropy frontier after logit shaping; they also achieve competitive quality and diversity for molecule generation.Source: §1 · p. 2
• Sampling a category from a diffused SDM simplex state has the same marginal as Discrete Diffusion, while SDMs reduce to Discrete Diffusion at high temperature and a deterministic path with Gaussian fluctuations at low temperature.The paper describes the inverse Dirichlet concentration as a temperature.Source: §1 · p. 2

Discrete Diffusion Models

• Discrete Diffusion models corrupt categorical data along marginals bridging pdata to a reference prior π (Sohl-Dickstein et al., 2015; Austin et al., 2021; Campbell et al., 2022; Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2025).The generative process applies compatible reverse transitions from the prior to recover samples from pdata.Source: §2 · p. 3
• Recent work uses learned bridge transitions based on xθ(t, xt), obtained by maximizing an expected Evidence Lower Bound (ELBO) (Gourevitch et al., 2026).These approximate transitions replace marginal transitions while remaining compatible with the forward process.Source: §2 · p. 3

Dirichlet Flow Matching

• Dirichlet Flow Matching extends Flow Matching to the simplex, connecting lifted data to a uniform Dirichlet prior through a probability path and a learned velocity field (Stark et al., 2024; Lipman et al., 2023).The velocity field is trained through cross-entropy to approximately follow the path from data at t = 0 to the prior at t = 1.Source: §2 · p. 3

3.1 PROBABILITY PATH AND BRIDGE

• The forward path uses a Dirichlet corruption with concentration c_t(α_t P_0 +(1−α_t)π), equivalently sampled as a random convex mixture of data and Dirichlet noise.The reference prior π should have full support; otherwise, observing a token outside its support would reveal that token before t=1.Source: §3.1 · pp. 3–4
• As concentration varies, the path connects discrete diffusion at the high-temperature limit to concentration around the path mean as ε→∞.For c_t=ε/(1−α_t), ε→0 concentrates conditional samples on simplex vertices; ε→∞ concentrates them around α_tP_0 +(1−α_t)π.Source: §3.1 · p. 4
• The backward bridge preserves path compatibility, while churn κ controls reverse-step randomness; under c_t=ε/(1−α_t), κ=0 yields the exact deterministic DDIM rule.The bridge is constructed to satisfy the backward compatibility condition between times s and t.Source: §3.1 · pp. 4–5

3.2 TRAINING AND INFERENCE

• At inference, a denoiser estimates the categorical posterior over simplex vertices, and its prediction conditions each backward transition; training minimizes cross-entropy.With uniform weighting over training times, the loss corresponds to a negative ELBO.Source: §3.2 · p. 6
• For large vocabularies, the denoiser can embed a simplex state using its most likely token rather than computing its expected embedding.The expected-embedding alternative multiplies the simplex state by the token embedding matrix.Source: §3.2 · p. 6

4.1 TEMPERATURE LIMITS

• At high temperature, SDM states concentrate on simplex vertices, and with κ = 0 the reverse bridge transitions between current and clean vertices, recovering a discrete-diffusion interpretation.The limiting reverse transition remains supported on simplex vertices.Source: §4.1 · p. 6
• At low temperature, randomness disappears and the forward state interpolates deterministically between P0 and π; this is only a first-order limit as ε → ∞.The paper notes a Gaussian fluctuation limit for this regime in Section E.7.Source: §4.1 · p. 6

4.2 CONCENTRATION SCHEDULE VIA NORMALIZED VARIANCE

• With a uniform prior, the concentration schedule can be parameterized by ν_t, the fraction of maximal total variance, placing the schedule on a bounded, interpretable scale.The section defines total variance as Tr[Cov(P_t | P0)] and uses ν_t to set its fraction of the bound.Source: §4.2 · p. 7
• At t = 1, c1 = N induces a uniform simplex distribution, while larger c1 concentrates around π; for t < 1, large c_t collapses P_t onto a mean moving toward the clean-token vertex.Total variance approaches its upper bound for vertex collapse and vanishes as c_t → ∞ when P_t collapses onto its mean.Source: §4.2 · p. 7

5.1 REASONING ON SUDOKU

• With SC, SDMs reach 99.1% Sudoku accuracy, within 0.2 points of the best overall result and above the continuous baselines.SDMs are the strongest continuous method and match discrete diffusion models; they also outperform Dirichlet Flow Matching by a wide margin (76.7%). Without SC, increasing κ from 0 to 1 raises accuracy from 73.9% to 88.4%.Source: Table 1 · p. 7, §5.1 · pp. 7–8

5.2 CODE GENERATION ON TINYGSM

• At 512 steps, expected-embedding SDMs without SC reach 45.8% at T=1 (vs. UDM + PC 36.8%) and 49.0% at T=0.1 (vs. MDM + SC 45.8%).Argmax + SC SDMs perform best at T=0.1 from 32 NFE onward, reaching 57.0% at 8k NFE; without SC, SDMs also outperform spherical flows without SC (45.8% vs. 32.4%) at 512 NFE and T=1.Source: §5.2 · p. 8
• Adaptive scheduling improves expected-embedding SDM accuracy by 9–11 points at 512 steps and 18–19 points at 64 steps, but does not help Argmax inputs.SDMs perform best with κ=1 (45.8% vs. 12.6% at κ=0, T=1, 512 steps); full-sequence SDMs also outperform block-autoregressive generation with block size 32 at matched NFE.Source: §5.2 · p. 8
• At 8 NFE, distilled SDMs solve 32.1% of GSM8K problems at T=1, versus 21.4% for IDLM at 128 NFE and about 6% for undistilled SDMs at 8 NFE.Distilled accuracy rises to about 40% at 128 NFE before plateauing, while AST diversity falls to 17.5 from 35.3 for undistilled SDMs at 512 NFE; auto-guidance improves accuracy in all tested settings.Source: §5.2 · p. 8

5.3 LANGUAGE MODELING ON OPENWEBTEXT (OWT)

• SDMs achieve an OWT GenPPL/Entropy trade-off of 17.0/5.46, close to real validation data, after a local frequency penalty.SDMs are the only models reported to benefit from the local frequency penalty; sequence frequency penalties benefit all tested model types.Source: Figure 3 · p. 9, §5.3 · p. 9

• SDMs maintain continuous distributional belief states throughout generation, avoiding information collapse even without Self-Conditioning, and use a flexible DDIM-type sampler that does not discretize an ODE or SDE.Self-Conditioning remains complementary to SDMs.Source: §6 · p. 9
• SDMs are competitive on unconditional text generation and molecular synthesis and state-of-the-art on coding, but experiments use only small models, leaving their validity and viability at scale open.Further experimentation is needed on large-scale language generation tasks.Source: §6 · p. 9
• SDMs suggest there is no strict dichotomy between discrete and continuous diffusion language models: Discrete Diffusion Models arise at the high-temperature limit, while an Argmax input lets the denoiser process discrete sequences within simplex-based generation.With Argmax input, the generative process itself remains continuous on the simplex.Source: §6 · p. 9

AI USE STATEMENT

• AI tools assisted with manuscript writing, proof derivation, and code, while agentic tools were used to run experiments.Source: AI USE STATEMENT · p. 10
• The authors state that the main ideas were introduced without AI, and that result interpretation and the manuscript’s first writing were done without LLM assistance.Source: AI USE STATEMENT · p. 10
• TinyGSM was fully generated by GPT-3.5 (Liu et al., 2023a), and the authors take responsibility for the paper’s content, claims, citations, and conclusions.Source: AI USE STATEMENT · p. 10

Theoretical Claims and Proofs

• Complete proofs for Propositions 3.1–3.3 and 4.1, extended theoretical results, and explicit mathematical assumptions are provided in Sections E, F, and G.Extended results include Gaussian fluctuation limits, the variational lower bound, and Simplex DMD distillation.Source: REPRODUCIBILITY STATEMENT · p. 10

Algorithms and Pseudocode

• Four self-contained algorithms cover training, inference, accelerated Marsaglia–Tsang Gamma sampling, and Simplex Distribution Matching Distillation, with multi-token and classifier-free guidance extensions.The extensions are documented in Sections C.1 and C.2; Gamma sampling is also described in Section D.Source: REPRODUCIBILITY STATEMENT · p. 10

Model Architectures and Hyperparameters

• The paper specifies network configurations, optimizer parameters, adaptive time schedules, denoiser input parameterizations, and variance schedules across Sections J.1, J.2, J.4, J.5, J.7, and J.8.Specifications include Diffusion Transformer dimensions and attention heads, Adam/AdamW settings, warmup, EMA decay, and gradient clipping.Source: REPRODUCIBILITY STATEMENT · p. 10

Experimental Details and Full Sweeps

• Full ablations and hyperparameter sweeps appear in Section K and Tables 21–23 and 29–32, including exact mean and standard deviation values across random seeds.Source: REPRODUCIBILITY STATEMENT · p. 10

B.1 TRAINING

• Training samples a data token and time, forms Dirichlet corruption parameters from the token’s one-hot vector and prior, then optimizes cross-entropy from the model’s clean-token prediction.Time is sampled uniformly by default, and the noisy simplex state is drawn from the resulting Dirichlet distribution.Source: §4 · p. 23, §2 · p. 23, §1 · p. 23

B.2 INFERENCE

• Inference initializes from a scaled prior, repeatedly samples a clean token prediction and reverse-transition variables to update the simplex state, then samples the endpoint token.The reverse transition includes Beta- and Dirichlet-distributed variables; the algorithm also exposes a churn parameter.Source: §4 · p. 23

C.1 EXTENSION TO MULTIPLE TOKENS

• For multiple tokens, forward and reverse kernels factorize across positions, while each position’s clean-token prediction can use the full noisy sequence as context.The extension applies the single-token reverse transition independently at each position and retains the sequence-level cross-entropy objective.Source: §C.1 · p. 24

C.2 CLASSIFIER-FREE GUIDANCE

• Classifier-free guidance combines unconditional and conditional predictions into a guided predictor, using guidance scale 0 for unconditional sampling, 1 for conditional sampling, and values above 1 for guidance.Training drops the condition on sampled examples, and inference substitutes the guided prediction into the reverse transition.Source: §C.2 · p. 25

D FAST SAMPLING OF DIRICHLET RANDOM VARIABLES

• The Dirichlet sampler uses Gamma variables and an exact Marsaglia–Tsang implementation that avoids vmap-based key generation; the illustrated PRNG pattern achieves a 3.8x speedup at N = 10^6 and d = 1.The stated motivation is to reduce the memory overhead of creating many keys.Source: D FAST SAMPLING OF DIRICHLET RANDOM… · p. 25, Figure 4 · p. 27

E.4 A REPRESENTATION OF Pt

• Exact DDIM makes each state a random convex combination of successive posterior clean-state draws and initial noise; plug-in transitions need not have this exact representation.The weights follow a Dirichlet distribution, motivating interpretation of the state as a belief-memory state.Source: §E.4 · p. 28

E.5 ILLUSTRATION AND TEMPERATURE LIMITS FOR PROPOSITION E.1

• Figure 5 shows the weight mean profile is independent of ε, alongside ±1 standard-deviation bands for ε = 0.01, 1, and 100 and indices i = 0, 5, 10, and 15.The figure visualizes the intrinsic Self-Conditioning weights from Proposition E.1.Source: Figure 5 · p. 29, §E.5 · p. 29
• As ε → ∞, the weights converge to a deterministic vector; as ε → 0, they converge to a categorical draw, recovering Discrete Diffusion behavior.At ε → 0, only one component contributes; at ε → ∞, clean-state draws are mixed deterministically.Source: §E.5 · p. 30
• Figure 6 illustrates that samples concentrate on simplex vertices as ε → 0, while the conditional mean remains independent of ε.The displayed conditional mean is α_tP_0 + (1 − α_t)π.Source: Figure 6 · p. 31

E.6 PROOF OF PROPOSITION 4.1

• As ε→0, SDMs recover vertex-supported discrete-diffusion behavior, while ε→∞ concentrates around deterministic interpolation; churn κ=0 trusts current beliefs and κ=1 gives the independent bridge.In the intermediate regime, SDMs maintain continuous beliefs during sampling.Source: Figure 7 · p. 32, §E.6 · p. 32

E.7 GAUSSIAN FLUCTUATIONS

• At scale sqrt(ε), forward-state fluctuations around the deterministic path converge to Gaussian, and projected covariance eigenvalues remain bounded between min_i π_i and 2.The fluctuation result assumes strictly positive prior components.Source: §E.7 · pp. 33–34

F VARIATIONAL LOWER BOUND

• The fixed-grid inference procedure admits a cross-entropy ELBO, while retaining the full reverse mixture yields a tighter but generally intractable mixture-KL bound.Sampling a time uniformly from the inference grid and then sampling the corresponding forward state gives an unbiased estimator of the negative ELBO divided by the grid size.Source: F VARIATIONAL LOWER BOUND · pp. 35–36

G.1 SIMPLEX DISTRIBUTION MATCHING DISTILLATION

• The multistep simplicial DMD method trains a generator to match posterior reverse distributions while producing outputs on the simplex.The generator is not required to output simplex vertices during distillation.Source: §G.1 · p. 36
• Stop-gradient techniques and a Dirichlet analogue of Tweedie’s identity yield a tractable surrogate for the valid but intractable distribution-matching loss, with identical gradients.The Dirichlet identity relates posterior expectations to derivatives of the observation density.Source: §G.1 · p. 37
• Simplex DMD derives a tractable equivalent loss with the same gradients; Algorithm 4 is exact with exact denoisers and valid pathwise derivatives, but learned denoisers yield approximate gradients.Source: §G.1 · p. 38

2. Teacher forward to time s

• Simplex DMD’s surrogate gives the exact KL gradient, including with stop-gradient, whereas IDLM’s discarded high-variance indirect gradient lacks a well-defined-objective guarantee (Li et al. (2026); Hoogeboom et al. (2026)).The Simplex DMD gradient is derived from the KL objective; discarding IDLM’s indirect gradient is an approximation without that guarantee.Source: §2 · p. 40
• Because the simplex forward map is differentiable via Γ reparameterization, its Jacobian is computed exactly by autodiff, unlike discrete IDLM’s biased approximations (Li et al. (2026)).Source: §2 · p. 40

H.1 GENERAL OVERVIEW

• The closest prior approach, Dirichlet Flow Matching (Stark et al. (2024)), shares the forward process but uses flow-based inference, while SDM uses DDIM-like inference and a temperature-parameterized process.Source: §H.1 · p. 41

H.2 SIMPLEX RELAXATION FOR DISCRETE DIFFUSION

• Unlike Sakurai et al. (2026)’s relaxation, whose sampling can be interpreted as a discrete backward transition with remasking, SDM tracks simplex states throughout generation and samples only at the terminal state.Sakurai et al. (2026) use a different training loss; SDM uses a simple cross-entropy loss.Source: §H.2 · p. 42

I.1 DISCRETE DIFFUSION MODELS

• Discrete diffusion baselines commonly use absorbing [MASK] or uniform priors, while prior work also studies mixtures of these priors and data-dependent priors.Source: §I.1 · p. 43
• Discrete-diffusion ancestral sampling applies reverse transitions along a time grid, and samplers can temperature-scale denoiser logits, with T = 1 unchanged and T approaching 0 yielding greedy decoding.For SDMs, the scaled prediction replaces the denoiser prediction in the reverse transition.Source: §I.1 · p. 43

I.2 PREDICTOR-CORRECTOR SAMPLER

• Predictor-corrector sampling predicts a clean sample and re-noises it at each step, allowing masked discrete-diffusion models to revise earlier token choices.Each step discards the current noisy state except through the clean-sample prediction; unmasked tokens may therefore be masked again in masked diffusion models.Source: §I.2 · p. 44

I.3 SELF-CONDITIONING AND LOOPHOLING

• Self-conditioning addresses discrete diffusion’s loss of denoiser uncertainty by carrying a continuous latent between sampling steps alongside the discrete state.With probability 0.9, training uses a no-gradient first pass to produce a latent and a second pass to compute the loss; SC models are trained for about 0.77 of the steps to match FLOPs. Sampling passes each step’s latent to the next without an extra forward pass.Source: §I.3 · pp. 44–45

I.4 DIRICHLET FLOW MATCHING

• Dirichlet Flow Matching (DFM) extends Flow Matching to the simplex with full-support paths, but its deterministic sampler differs from SDM’s, which remains stochastic even when κ = 0.Stark et al. (2024) train a denoiser with cross-entropy; their sampling marginalizes the conditional and replaces the true posterior with the learned one. Flow Matching avoids optimizing through expensive ODE simulation, though sampling still integrates an ODE.Source: §I.4 · pp. 45–46

J.1 TIME SAMPLERS AND SAMPLING SCHEDULES

• Adaptive time sampling draws training times from a fitted loss-change profile, while inference schedules may be linear, cosine, or adaptive; the adaptive grid requires adaptive sampling and adds no inference tuning or computation.The adaptive grid allocates more denoising steps where loss changes fastest, matching the noise levels emphasized during adaptive training.Source: §J.1 · p. 47

J.2 DENOISER INPUT FOR SDMS

• Argmax input uses a table lookup, yet the sampler still transitions on the full simplex state; expectation input retains belief-state information but adds about 29% to TinyGSM forward-pass FLOPs.Self-Conditioning provides a rich summary of the previous prediction and complements Argmax input.Source: §J.2 · p. 48

J.3 DIRICHLET FLOW MATCHING BASELINE

• DFM was adapted with an adaptive training-time sampler and numerical methods for large vocabularies, including a geometric grid around the prior mean and complementary Beta-CDF calculations for improved precision.The implementation also uses numerical integration to avoid underflow and overflow in the velocity calculation.Source: §J.3 · pp. 48–49
• 0.03 KL-divergence on the 40-category toy task approximately matched Stark et al. (2024, Figure 4), and three sampling changes enabled DFM at 50k vocabulary scale.The DFM baseline was reimplemented and extended for the paper’s datasets and larger vocabularies.Source: §J.3 · pp. 49–50

J.4 SUDOKU

• Sudoku evaluation uses 200k training and 5k validation puzzles with exact-match accuracy; training and validation share no complete grids, and the larger split addresses observed mild overfitting.Only target tokens are corrupted during training, and inference denoises the target half from a clean puzzle prompt.Source: §J.4 · p. 50

J.5 TINYGSM

• TinyGSM models are evaluated zero-shot on 1,319 GSM8K problems by executing generated programs, while AST diversity measures structural variation among sampled programs and can be restricted to correct solutions.The TinyGSM dataset contains approximately 11.8M synthetic problems, and the evaluation uses the 49k-token SmolLM tokenizer.Source: §J.5 · pp. 50–51

J.6 OPENWEBTEXT

• OpenWebText evaluation corrupts and denoises 1,024-token sequences from scratch, measuring sample quality with GPT-2-large Generative Perplexity and unigram entropy.Diffusion models use 64 sampling steps; the autoregressive model generates token by token in 1,024 steps.Source: §J.6 · p. 51

J.7 LANGUAGE UNDERSTANDING

• Language-understanding benchmarks ARC-easy, PIQA, and HellaSwag measure accuracy in selecting the correct continuation from multiple choices, unlike the sample-quality metrics used for Sudoku, TinyGSM, and OpenWebText.Source: §J.7 · pp. 51–52
• Language-understanding comparisons include GPT-2, retrained LLaMA (Touvron et al., 2023; von Rütte et al., 2025), GIDDs (von Rütte et al., 2025), MDMs (Deschenaux & Gulçehre, 2025), and PGMs (Deschenaux et al., 2026b).Continuation scores are normalized by continuation byte length, and accuracy is averaged over each task’s test set.Source: §J.7 · p. 52
• Because training noised whole 1,024-token sequences but evaluation noised only continuations, the authors trained a 20,000-iteration adaptation run using clean contexts and noised continuations.They also note that Simplex Diffusion retains information about the initial sample, motivating a regularization term that emphasizes continuation scoring over whole-text plausibility.Source: §J.7 · p. 52

J.8 UNCONDITIONAL MOLECULAR GENERATION

• The Constant-linear schedule (ν0 = 0.4, ν1 = 0.75, ℓ = 0.2) achieved the best molecular-generation Quality and was used for all reported Simplex Diffusion evaluations.It was selected from five concentration schedules after checkpoint selection using Quality measured on 1,000 generated molecules.Source: §J.8 · p. 53
• Molecular-generation evaluation reports validity, uniqueness, diversity, and Quality over 1,000 generated molecules for each of three sampling seeds, as mean ± standard deviation.Quality is the percentage of samples that are valid, unique, and drug-like, with drug-likeness defined by QED ≥ 0.6 and SA ≤ 4.Source: §J.8 · p. 53
• Molecular-generation diffusion models were evaluated with 32, 64, 256, or 1,024 steps and linear, cosine, or adaptive schedules; detailed results use 256 steps.The adaptive schedule is described in Section J.1.Source: §J.8 · p. 53
• Confidence-based sampling commits the most confident positions directly to predicted tokens, and the authors report benefits for Simplex Diffusion despite its resulting samples being out of distribution.The remaining positions use the standard reverse update; the confidence-based strategy is used without retraining.Source: §J.8 · pp. 53–54

K.1 SUDOKU

• With SC, Sudoku SDMs reach 99.1%, near MDM with SC at 99.3%; without SC, they reach 88.4%, outperforming DFM (76.7%) and FLMs (75.3%) and matching S-FLMs (87.3%).Without SC, concentration schedules reach 86–92% at κ=1, while none exceeds 79.4% at κ=0; with SC, all three churn values reach 98.6–99.2%. The adaptive time sampler improves ancestral sampling without SC but raises variance with SC, so SDMs use the uniform time sampler in the main text.Source: §K.1 · p. 54, Table 2 · p. 54

K.2 TINYGSM

• At T=0.1 and 512 steps, TinyGSM SDMs with κ=1 reach 50.9%, above the best UDM with PC at 45.5%; expectation-input SDMs reach 49.0% with the adaptive grid.Accuracy generally rises with churn, though the trend is not monotone in every configuration; the adaptive grid brings its largest expected-embedding gains at small step budgets.Source: Figure 9 · p. 55, §K.2 · pp. 55, 57
• Without SC, TinyGSM SDMs beat Spherical Flow with PC at 64 NFEs (32.7% vs. 26.9%) and 512 NFEs (45.8% vs. 32.4%); full-sequence SDMs also outperform blockwise generation at every NFE budget.The reimplemented DFM performs worse than SDMs; blockwise generation uses blocks of 32 tokens from left to right.Source: §K.2 · pp. 55, 57, 58, Figure 12 · p. 57, Table 4 · p. 58

K.3 DISTILLATION ON TINYGSM

• 32.1% pass@1 with 8 steps exceeds IDLM’s 21.4% with 128 steps; all 36 configurations score 27.4–32.1% (median 28.9%).The 32.1% is the best-of-sweep result selected on the GSM8K test set without a validation split. Distillation trades diversity for speed: at 8 NFEs, AST diversity is 17.5, versus 35.3 for the undistilled SDM at 512 NFEs and 36.7 for AR.Source: §K.3 · p. 58

K.4 AUTO-GUIDANCE ON TINYGSM

• Auto-Guidance improves TinyGSM accuracy across the investigated frameworks, with early-to-intermediate checkpoints (20k–50k) consistently achieving peak accuracy.It guides the converged model away from an earlier unconverged checkpoint; its effect is strongest at low churn and high temperature, while checkpoints near convergence yield accuracy closer to the unguided baseline.Source: §K.4 · pp. 58, 60, Figure 14 · p. 60

K.5 OPENWEBTEXT

• OpenWebText inference interventions combine temperature scaling and nucleus sampling with frequency penalties, while the most influential intervention is the sequence-level frequency penalty.The diffusion temperature annealing schedule is not claimed to be optimal; the local-frequency intervention is reported only for SDMs because it does not improve the UDM Pareto frontier and does not change MDM predictions without remasking.Source: §K.5 · pp. 61–62
• On OpenWebText, the autoregressive Pareto frontier scored better than real data, raising questions about generative frontiers as a meaningful language-generation benchmark.The appendix also reports intervention effects on four model families and matched-entropy and matched-GenPPL hyperparameter selections.Source: §K.5 · p. 62

K.6 LANGUAGE UNDERSTANDING

• For language understanding, the adapted model outperformed the baseline, supporting the claim that the baseline is out of distribution when only the continuation is noised.The passage points to a separate table for the main benchmark comparison but supplies no benchmark values.Source: §K.6 · p. 63

K.7 UNCONDITIONAL MOLECULAR GENERATION

• The preferred SDM sampling-time schedule varied with sampling budget, sampler, and evaluation metric, so the authors selected the adaptive schedule as an overall trade-off.The schedule ablation compared standard and confidence-based sampling across linear, cosine, and adaptive grids.Source: §K.7 · p. 65, Figure 20 · p. 68
• Masked and uniform diffusion achieved higher Quality but lower Diversity than SDM; with confidence-based sampling, SDM matched UDM when time conditioning was removed.The results illustrate a trade-off between Quality and Diversity across diffusion methods.Source: §K.7 · pp. 65–66
• The Simplex Diffusion Pareto frontier leaned toward higher Quality and Diversity, with many configurations in the high-Diversity region, but it did not attain the highest Quality.The frontier compared sampling configurations varying seeds, step counts, temperatures, churn, and top-fraction power.Source: §K.7 · pp. 66–67, Figure 23 · p. 69
• Time conditioning raises selected confidence-based Quality from 85.87% to 87.23% for SDM with an adaptive grid, and from 86.03% to 91.97% for UDM.Source: §K.7 · p. 68
• All reported diffusion configurations exceeded the AR baseline in Quality; time-conditioned SDM surpassed published GenMol in Quality and Diversity, though protocols differ, while maximum Quality still trails masked diffusion.The authors identify pushing the SDM Pareto frontier and closing the maximum-Quality gap as future work.Source: §K.7 · p. 69

L ADDITIONAL OWT SAMPLES

• Five non-cherry-picked unconditional 1024-token samples are provided from SDMs trained on OWT.Tables 16–20 show the samples, with each table caption identifying one sample.Source: L ADDITIONAL OWT SAMPLES · p. 69, Table 16 · p. 71, Table 17 · p. 72

M.1 SUDOKU

• Sudoku SDM tables report accuracy in 180 steps across sampling schedules for churn κ values 0.0, 0.2, and 1.0.Values are reported as mean and standard deviation over five seeds.Source: Table 21 · p. 76, Table 22 · p. 77, Table 23 · p. 78

M FULL RESULT TABLES

• Sudoku baseline tables compare model accuracy across sampling schedules in 180 steps, covering autoregressive, discrete diffusion, flow-matching, and hyperspherical-flow methods.The listed methods include discrete absorbing models with CE and ELBO losses, discrete uniform diffusion, predictor-corrector, and self-conditioning.Source: Table 24 · p. 79
• TinyGSM baseline tables report accuracy across sampling schedules for temperatures 1.0 and 0.1 and budgets of 512 and 64 steps.The evaluated baselines include autoregressive, discrete absorbing and uniform diffusion, predictor-corrector, self-conditioning, FLM, and S-FLM.Source: Table 25 · p. 80, Table 26 · p. 81, Table 27 · p. 82

M.2 TINYGSM

• TinyGSM SDM tables report accuracy across noise-concentration and sampling schedules, churn settings, and expectation, argmax, and self-conditioned argmax sampling.The four tables cover temperatures 1.0 and 0.1 at 512 and 64 steps; all SDMs use adaptive time sampling for training.Source: Table 29 · p. 83, Table 30 · p. 84, Table 31 · p. 84
• TinyGSM multi-sample evaluation reports accuracy and AST structural diversity for five samples per problem across step counts and temperatures.The table covers NFE values 8, 64, and 512 and temperatures 0.1 and 1.0.Source: Table 33 · p. 86