PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Design Space of Continuous Diffusion Language Models

Oscar Davis, Daan Roos, Alice Bizeul, Amitis Shidani, Victor Turrisi, Louis Béthune, Pierre Ablin, Marco Cuturi

arXiv:2610.13834v1cs.LG

TL;DR

Evidence remains limited on which training choices matter for continuous diffusion language models and how embedding geometry interacts with noise schedules. The paper studies self-conditioning and embedding geometry, introducing DoubleCat and SemiCat alongside analyses of embedded methods. SemiCat attains discrete-diffusion-level performance on reported benchmarks, with strong likelihood and unconditional-generation results at scale.

  • Problem

    The mechanisms by which self-conditioning and noise schedules interact with training trajectories and learned embeddings in continuous diffusion language models remain unclear.

    Source: §1 · p. 1

  • Method

    The paper analyzes self-conditioning and embedding geometry, and introduces DoubleCat and SemiCat as interventions for trajectory mismatch and token identifiability.

    Source: §1 · p. 2, §4.3 · p. 11

  • Results

    SemiCat attains discrete-diffusion-level performance on reported benchmarks, outperforming discrete diffusion on four of six likelihood tasks at scale and surpassing one-hot flows in unconditional generation.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    DoubleCat recovers self-conditioning gains without an additional forward pass, while SemiCat reaches discrete-diffusion-level benchmark performance.

    Source: §1 · p. 2, §7 · p. 12

    SemiCat is harder to accelerate with typical continuous techniques; asynchronous schedules are discussed as a possible way to alleviate this issue.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion. Yet, unlike for continuous modalities such as images or videos, our grasp of the levers that matter when training CDLMs remains limited. In this work, we focus on two of these levers, namely self-conditioning and embedding geometry, and shed light on some of the mechanisms behind them, which in turn suggest cheaper and more effective designs. We begin by showing that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training. These gains can be recovered without an extra forward pass by a two-level interpolant, coined $\mathrm{DoubleCat}$, which improves TinyGSM accuracy more than threefold. Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians$\unicode{x2013}$an issue known as token identifiability. To address it directly, we introduce discrete reveals to $\mathrm{DoubleCat}$, yielding $\mathrm{SemiCat}$, which achieves discrete diffusion-level performance on TinyGSM, reaching $49.94\%$ with standard training hyperparameters and a tuned sampler. At the $1.7$B scale, $\mathrm{SemiCat}$ outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation. On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one. We briefly explore a range of further options, such as hybrid corruption processes and noise schedules, including asynchronous ones. We believe this exploration can help the community better understand and advance the design space of CDLMs.

1 Introduction

The paper examines how self-conditioning and embedding geometry shape continuous diffusion language models, proposing cheaper ways to address trajectory mismatch and token identifiability. SemiCat reaches discrete-diffusion-level performance on TinyGSM and performs strongly at scale.

1 Introduction

  • DoubleCat recovers self-conditioning gains without an additional model forward pass, improving TinyGSM accuracy from 4.42% to 16.00%.The two-level interpolant increases the density of otherwise under-sampled, inference-like states during training.Source: §1 · p. 2
  • Self-conditioning addresses a train/inference trajectory discrepancy by increasing training exposure to otherwise under-sampled regions.The paper links this mechanism to the greater density of inference-like points at training time, consistent with Shabalin et al. 2026.Source: §1 · p. 2
  • Discrete stochastic reveals yield SemiCat, which helps the model condition on parts of its context and improves token identifiability.SemiCat reaches the discrete diffusion baseline of about 33.74% on TinyGSM, with accuracy up to 32.90% before the tuned-sampler result.Source: §1 · p. 2
  • At 1.7B, SemiCat outperforms discrete diffusion on four of six likelihood-based evaluations and surpasses one-hot flows in unconditional generation.On TinyGSM, SemiCat reaches 49.94% with standard training hyperparameters and a confidence-based sampler.Source: §1 · p. 2
  • Cross-entropy spreads embeddings, producing a geometry whose decoding error rate resembles the one-hot case.The authors argue that training should update all embeddings jointly.Source: §1 · p. 2
  • The paper also explores asynchronous schedules, dynamic time allocation, hybrid methods, and acceleration as parts of the CDLM design space.The authors describe these explorations as avenues for further understanding continuous diffusion for discrete data.Source: §1 · p. 2

2 Background

The background introduces flow-based generative modelling and its adaptation to discrete language, then describes one-hot, embedded, hybrid, and self-conditioned approaches. It also outlines relevant schedules and Plaid.

2 Background

  • Flow-based generative modelling transports a simple starting distribution toward the data distribution along a probability path defined by an ODE.The model learns a velocity field, and sampling integrates that field.Source: §2 · p. 2
  • Flow-based language modelling adapts flow modelling to data distributed over discrete token sequences.One-hot flows represent tokens as one-hot vectors in R^|V| and use an endpoint parametrisation of the generating vector field.Source: §2 · pp. 2–3
  • Embedded methods map tokens through a trainable codebook into a space E of dimension d no greater than the vocabulary size.The embedding map extends to sequences element-wise, and W contains the token embeddings.Source: §2 · p. 3
  • One-hot flows default to a mixture time schedule combining a decoding-error-rate schedule with linear time.The mixture weight is 0.75, and the decoding-error-rate schedule makes uniform progress in decoding error rate.Source: §2 · p. 3
  • CANDI continuously evolves each sequence position until a randomly sampled reveal time, when it exposes the clean token as a one-hot encoding.FMLM+ instead begins with random token reveals and commits additional tokens during sampling based on model confidence.Source: §2 · p. 3
  • Plaid is a variational diffusion model for language modelling, trained by minimizing the negative evidence lower bound with learned schedules.Its encoder maps embedded inputs across time, and its prior is Gaussian.Source: §2 · p. 4
  • Self-conditioning feeds the denoiser its previous inference-step output and supplies that signal during training with probability psc.The method extends the denoiser with an additional input and uses a stop-gradient operator for the self-conditioning signal.Source: §2 · p. 4

3 Enhancing one-hot flows

One-hot flows face two problems: inference trajectories pass through states sparsely represented in training, and large Gaussian noise makes tokens hard to identify. DoubleCat and SemiCat address these problems with altered corruption and revealing mechanisms.

3.1 Trajectory ambiguity

  • Near t ≈1, one-hot flows train on concentrated Gaussians near clean endpoints, but inference follows a denoiser with non-zero entropy, creating a train–inference mismatch.Self-conditioning increases training density in likely inference regions; DoubleCat adds a simplex corruption process to imitate those states.Source: §3.1 · pp. 4–6
  • 16.00% TinyGSM accuracy with T = 0.2 is over 4× the one-hot-flow result; DoubleCat also lowers name errors by 18 points and raises valid programs by almost 19 points.Self-conditioning further improves DoubleCat to 20.77% at T = 0.5.Source: §3.1 · p. 6

3.2 Token identifiability

  • Large Gaussian noise obscures current tokens, so SemiCat adds random one-hot reveals that let the model condition on revealed inputs.The section attributes DoubleCat’s remaining name errors (30.02%) to token identifiability.Source: §3.2 · p. 6
  • 32.90% TinyGSM accuracy without self-conditioning reaches the discrete-diffusion level, while SemiCat reaches 49.94% with standard hyperparameters and a tuned sampler.SemiCat also has 10.39% runs with name errors and programs that run 89.23% of the time; on OpenWebText it dominates one-hot flows’ Gen. PPL/entropy frontier at all NFEs.Source: §3.2 · p. 7, Figure 3 · p. 7
  • SemiCat’s stochastic reveals and piecewise-continuous trajectories make it harder to accelerate with typical continuous techniques.Asynchronous per-position schedules are discussed as a possible way to alleviate this difficulty.Source: §3.2 · p. 7

4 Designing embeddings

The section examines how embedding geometry and self-conditioning affect continuous language models. It identifies one-hot flows as embedded models, finds cross-entropy can recover one-hot-like decoding error rates, and explores cheaper self-conditioning alternatives.

4 Designing embeddings

  • One-hot flows incur memory and computation costs from materialising an L × |V| tensor per batch example, motivating smaller embedded spaces.The section notes that modern language-model vocabularies can exceed 200k tokens.Source: §4 · p. 7

4.1 Your one-hot flow is secretly embedded

  • One-hot flows operate on a projected embedded space, with injected-noise covariance determined by WᵀW.The same observation applies to DoubleCat and SemiCat; direct embedded-space SemiCat training reaches the same end performance but is less stable due to a Cholesky decomposition.Source: §4.1 · p. 8

4.2 Your embedded flow is secretly one-hot

  • Ambient-sphere training underperforms Plaid with self-conditioning and has Gen. PPL beyond 200 at matched data entropy.The section attributes this to cross-entropy updating only batch-involved token embeddings and ignoring their geometry, unlike methods that update all embeddings jointly.Source: §4.2 · pp. 8–9
  • Ambient-sphere embeddings have DER curves indistinguishable from one-hot except for a small gap near t≈0.85.Cross-entropy spreads tokens apart, approaching the one-hot DER regime.Source: Figure 4 · p. 9, §4.2 · p. 9

4.3 Self-conditioning on embedded spaces

  • Plaid runs without self-conditioning worsen monotonically as embedding dimension d increases, consistent with a growing train/inference discrepancy.The proposed distortion mechanism does not cover that mismatch or inference-time self-conditioning.Source: §4.3 · pp. 9, 11
  • Stochastic-codebook noise is most effective at low temperatures but does not recover self-conditioning performance.The codebook uses Gaussian embeddings regularised to 1/d variance per coordinate and unit-norm means.Source: §4.3 · p. 10
  • 18.35% vs 11.98%: posterior-expectation conditioning improves TinyGSM accuracy over no self-conditioning; true self-conditioning at inference reaches 20.24%.The posterior expectation is deterministic given the input, so gains come from a better state representation, not added input information.Source: §4.3 · p. 10
  • Self-conditioning provides a more identifiable state representation on the convex hull of the embeddings, aiding token identifiability.The study replaces it at training time with the Bayesian posterior expectation as a cheaper proxy.Source: §4.3 · pp. 10–11

5 Scaling SemiCat

At 1.7B parameters, SemiCat is compared with one-hot flow and discrete diffusion baselines on likelihood-based benchmarks and unconditional generation. It leads on most reported likelihood evaluations and dominates one-hot flow on the reported generation frontier.

5 Scaling SemiCat

  • The likelihood-based evaluations use continuation cross-entropy as a proxy for likelihood rather than an ELBO.The comparison follows six multiple-choice question-answering benchmarks.Source: §5 · p. 11
  • SemiCat beats discrete diffusion by 8.4 points on ARC-E, 2.0 on BoolQ, 7.6 on PIQA, and 1.2 on SIQA.The models have 1.7B parameters and were trained on Nemotron for 2.1T tokens.Source: §5 · pp. 11–12
  • SemiCat trails discrete diffusion by 2.8 points on OBQA, slightly exceeds Duo and MDLM on RACE (+0.3), and beats one-hot flow on every MCQA task.The six evaluations are ARC-easy, BoolQ, OBQA, PIQA, RACE, and SIQA.Source: §5 · pp. 11–12
  • At all NFEs, SemiCat dominates the one-hot flow baseline on unconditional generation's Gen. PPL/Entropy Pareto frontiers.The comparison sweeps NFEs and temperature on 2048-token generations.Source: §5 · p. 12

6 Related Work

The related-work section traces continuous flow methods for language and describes parallel progress on self-conditioning and embedded-space designs. It situates the paper among advances in flow modelling, spherical flows, and learned or pretrained embeddings.

6 Related Work

  • Gaussian interpolation on one-hot encodings was introduced from a variational perspective in VFM and later used for self-distillation and steering in CFM/FMLM/DFM.Yoo et al. (2026) study self-conditioning through fixed-point iteration, while Shabalin et al. (2026) use a multi-modality lens.Source: §6 · p. 12
  • S-FLM and von Mises-Fisher flows first showed reasonable continuous-method TinyGSM performance using spherical paths, while LangFlow and ELF pursued simpler or learned embeddings.ELF found pretrained encoders more effective, and Meshchaninov et al. (2026) use a pretrained autoregressive model for strong embeddings.Source: §6 · p. 12
  • Additional references are provided in Appendix A.Source: §6 · p. 12

7 Conclusion

The conclusion links self-conditioning to trajectory ambiguity and token identifiability, and presents SemiCat as a hybrid that reaches discrete-diffusion-level performance on the reported benchmarks. It also situates the design space among related approaches and leaves further acceleration and interactions with the continuous toolbox for future work.

7 Conclusion

  • Self-conditioning addresses trajectory ambiguity and token identifiability, while DoubleCat and improved state representations recover substantial benefits at lower cost.The conclusion also notes that one-hot methods evolve over embedded spaces with anisotropic noise, while naively trained embedded methods recover one-hot decoding error rates.Source: §7 · p. 12
  • At scale, SemiCat outperforms discrete diffusion on four of six likelihood tasks and surpasses one-hot flows in unconditional generation.The conclusion identifies accelerating these methods with the continuous toolbox and understanding its interaction with design choices as future work.Source: §7 · p. 12

A Extended related work

  • Earlier geometric approaches flowed on the Fisher–Rao simplex or positive orthant on the sphere, but did not scale to larger vocabularies.Other explored directions include sequence compression and contextual embeddings, with the latter described as more computationally expensive at training time.Source: A Extended related work · p. 16
  • Related work on hybrid corruption processes frames discrete tokens as anchors for reverse-process jumps and identifies the CANDI-style hybrid underlying SemiCat as an identity-kernel case.The authors describe this framework as complementary to their Appendix E.5 variants.Source: A Extended related work · p. 16

B DoubleCat derivations

The appendices derive properties of DoubleCat, describe SemiCat sampling, contrast discrete diffusion variants, and explain how Plaid’s variational diffusion schedule is parameterized. Together, these passages give mathematical and procedural context for the paper’s methods and comparisons.

B DoubleCat derivations

  • The inference-trajectory prediction is a convex combination of denoiser predictions on the simplex, with entropy bounds following from concavity and Jensen’s inequality.The derivation also gives the token-posterior covariance as diag(q) − qq^T, with trace 1 − ||q||², and contrasts it with the Gaussian posterior covariance for log-concave continuous densities.Source: B DoubleCat derivations · pp. 16–17

C Extended SemiCat presentation

  • SemiCat sampling follows CANDI, while the tuned scheme adds DDIM-style renoising and confidence-based remasking.Remasking returns a position to a continuous state.Source: C Extended SemiCat presentation · p. 17

D.1 Discrete diffusion

  • Masked diffusion reveals tokens by unmasking and never revisits revealed tokens, whereas uniform-state diffusion can revise any token at any time.Uniform-state diffusion must both locate and correct errors, and its objective scores probability ratios rather than only the true-token probability.Source: §D.1 · p. 19

D.2 The VDM noise schedule

  • For Plaid, the fixed-denoiser bound depends on the schedule through its endpoints, while the intermediate shape is learned to reduce Monte Carlo-estimate variance.The schedule is parameterized through a monotone network for log SNR; equal SNR levels yield the same noisy input regardless of the time at which they occur.Source: §D.2 · p. 20

E.1 Hyperparameters and experimental setup

The experimental setup uses adapted diffusion-transformer architectures and reports training and evaluation conventions across OpenWebText, TinyGSM, and Nemotron. TinyGSM experiments also include a run with standard literature hyperparameters.

E.1 Hyperparameters and experimental setup

  • Models use the adapted diffusion transformer, with EMA evaluation weights except for the large Nemotron case, and are evaluated at the final checkpoint.OWT and TinyGSM use 8× H100 GPUs for training; Nemotron 1.7B uses 256× H100 GPUs.Source: §E.1 · p. 20
  • One-hot flows, including DoubleCat and SemiCat, use loss weighting inversely proportional to the square root of the prediction-to-target L2 distance.The passage attributes this weighting to prior work.Source: §E.1 · p. 20
  • OpenWebText uses 1024-token sequences and the GPT-2 tokenizer, while the TinyGSM dataset pairs 11.8M grade-school math questions with Python solution-generating traces.TinyGSM results include a final SemiCat run using standard literature hyperparameters.Source: §E.1 · pp. 20–21

E.2 DoubleCat ablations

The ablations examine DoubleCat’s simplex prior and schedule, compare self-conditioning settings, and provide additional OpenWebText Pareto frontiers. The reported prior ablation favors a uniform simplex prior among the listed choices.

E.2 DoubleCat ablations

  • The ablations cover prior and schedule choices for DoubleCat, plus self-conditioning on and off for DoubleCat and one-hot flow.Accuracy is reported per sampler, with the final two columns giving each model’s best value over those samplers.Source: §E.2 · pp. 21–22, Table 7 · p. 22, Table 8 · p. 22
  • The uniform simplex prior consistently outperforms the barycentre and uniform-vertex choices on accuracy in the DoubleCat ablation.The ablation uses NFE 256 and sets the inner schedule to κ_t = t².Source: Table 6 · p. 22

E.3 OpenWebText Pareto Frontiers

  • The OpenWebText Pareto-frontier figures report GPT-2-measured generation perplexity for 1024-token sequences; one figure spans Plaid embedding dimensions and the other fixes d = 16.The passage identifies the plotted scope but does not state a comparative outcome.Source: Figure 6 · p. 23, Figure 7 · p. 23, §E.3 · p. 23

E.4 Corruption as data augmentation

This section tests corruption and hybrid-flow variants, examines an embedded-space account of SemiCat, and explores distillation. The results identify trade-offs in diversity, discrete reveals, and sampling balance.

E.4 Corruption as data augmentation

  • Corrupting interpolant inputs while keeping clean loss targets lowers OWT unconditional-generation diversity; comparable Gen. PPL requires very low constant ρ, where lower PPL signals entropy collapse.The comparison uses the usual mixture schedule and reports Gen. PPL with token entropy.Source: §E.4 · p. 23, Table 9 · p. 24

E.5 General hybrid corruption processes

  • α-Cat improves TinyGSM accuracy as α rises through 0.75, but α=1 becomes a leave-one-out denoiser and α>0 relies on discrete reveals.α=0 recovers CANDI; variants with α>0 partly defy the purpose of hybrid methods.Source: §E.5 · pp. 23–24, Table 10 · p. 24

E.6 Acceleration via D-MMD

  • D-MMD-distilled SemiCat is competitive with FMLM+, tending toward higher accuracy at 1 and 32 steps, while FMLM+ stays balanced across step counts.The reported results use naive sampling; the table describes students evaluated at 32 NFE.Source: §E.6 · p. 24, Table 11 · p. 25

F.1 Reproducing SemiCat’s statistics on an embedded space

  • Embedded-space SemiCat reproduces normal SemiCat’s final performance, but Cholesky’s positive-definiteness requirement makes training less stable.The construction matches the covariance structure using a Cholesky decomposition of WᵀW.Source: §F.1 · p. 25

F.2 Embedding-dependent schedules

  • Cross-entropy-spread embeddings approximately inherit one-hot DER under the unit-norm, roughly uniform-sphere approximation.The pairwise comparisons recover the one-hot expression up to O(ρ_ij) corrections.Source: §F.2 · pp. 26–27

F.3 Cross-entropy, mean squared error and the embedding geometry

These sections analyze how cross-entropy and MSE shape embeddings, how Plaid incorporates self-conditioning, and how codebook geometry and time schedules affect denoising and decoding.

F.3 Cross-entropy, mean squared error and the embedding geometry

  • Cross-entropy spreads embeddings by pushing the true token away from its predicted mean; MSE pulls it toward that mean and can move confused tokens toward the truth, but alone collapses at W=0.The gradient account is informal because a trained network only approximates the posterior.Source: §F.3 · p. 28

F.4 Plaid self-conditioning incorporation

  • Plaid passes the usual input and self-conditioning signal into the diffusion transformer through an effective input matrix twice the original size.The matrix grows from d × h to 2d × h.Source: §F.4 · p. 28

F.5 Embedded self-conditioning: beyond Euler steps

  • After about 80k steps, input and self-conditioning projections align at cosine similarity 0.9; after a roughly 30k-step plateau, loss, performance, and alignment improve as similarity falls to 0.6.The authors interpret the later change as evidence that the model learns more than a mere Euler-like step.Source: §F.5 · p. 28, Figure 8 · p. 28

F.6 Self-conditioning and token identifiability

  • For unit-norm codebooks, MMSE depends on SNR, token distribution, and Gram matrix—not dimension explicitly—while normalized-input distortion scales linearly with dimension.The minimum attainable MMSE is non-increasing with dimension because the set of realizable Gram matrices grows with dimension.Source: §F.6 · pp. 29–30

G.1 Entropic schedule

  • The entropic schedule advances linearly in expected remaining entropy, uses a one-time lookup table, closely tracks the mixture schedule, and matches its TinyGSM performance.Its Gen. PPL/entropy curve has an outlier near 3 nats, especially at low NFE, despite the same TinyGSM performance.Source: §G.1 · p. 31

G.2 Asynchronous schedules

The sections explore asynchronous per-position timing and adaptive allocation of training times. Asynchronous schedules improve several matched settings, while the allocation procedure directs training toward changing conditional-entropy regions.

G.2 Asynchronous schedules

  • Discrete reveals create stochastic discontinuities that complicate continuous diffusion tools; adaptive state-dependent timing requires two denoiser passes or schedule-agnostic training with uninformative inference statistics.The cited flow-map formalism assumes deterministic and continuous flow maps.Source: §G.2 · p. 31
  • Each position gets a local time from a global time via a monotone, approximately mean-preserving schedule; k→0 yields deterministic left-to-right reveal, while k→∞ shares time.The figure shows the local times and their left-to-right spatial profile at k=1.Source: §G.2 · pp. 31–32, Figure 11 · p. 32

G.3 Adaptive allocation of training density

  • InfoNoise adaptation changes the training-time sampling distribution, not loss weighting, and allocates samples in proportion to the rate of change in estimated conditional entropy.Cross-entropy serves as a proxy for conditional entropy; regions where it changes rapidly receive more training samples.Source: §G.3 · pp. 32–33

G.2 Asynchronous schedules

  • Asynchronous times improve accuracy, valid programs, and name errors in three of four matched settings, but all settings remain well below SemiCat.The exception is the linear schedule with InfoNoise; InfoNoise raises accuracy almost uniformly, but lowers valid programs for the mixture schedule.Source: §G.2 · p. 31, Table 12 · p. 33

G.3 Adaptive allocation of training density

  • The adaptive time-density allocation is bounded below and above so every corruption level retains training signal and no region receives excessive probability mass.The reported thresholds are π_min=0.1 and π_max=10.Source: §G.3 · p. 33
Loading 2610.13834v1…