PaperlayerView exact revision on arXiv

Source-linked AI summary

Token Time Continuous Diffusion for Language Modeling

Parikshit Bansal, Sujay Sanghavi

arXiv:2607.14106v1cs.CLcs.AI

TL;DR

Discrete diffusion language models can lose quality at high speedups because independently denoising multiple tokens misses their joint distribution. TTCD instead uses continuous, deterministic token evolution with per-token times, outperforming discrete methods in high-speed unconditional and conditional generation while remaining competitive on Sudoku.

  • Problem

    Discrete diffusion language models suffer a factorization problem at high speedups because independently sampled token marginals do not represent the true joint distribution.

  • Method

    TTCD deterministically evolves token embeddings from Gaussian noise to clean tokens in continuous space, assigning different time trajectories to tokens.

  • Results

    TTCD outperforms the best known discrete method at high-speed unconditional and prefix-conditioned generation and is competitive on Sudoku solving.

  • Takeaways & Limitations

    Token-time continuous diffusion is robust to inference-time settings and naturally supports prefix-conditioned generation at high speed.

  • Takeaways & Limitations

    The token-time mapping assumes all tokens start at noise, end at data, and preserve rank ordering throughout denoising.

Abstract

from arXiv · show

In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.

1 Introduction

TTCD addresses the factorization errors of discrete diffusion language models during high-speed generation by deterministically evolving token embeddings from Gaussian noise to clean tokens in continuous space. It introduces per-token times and rank-based denoising so tokens can progress at different rates and influence one another differently.

  • Discrete diffusion language models that denoise multiple tokens per step sample from product marginals rather than the true joint distribution.This factorization problem is especially relevant when generation is substantially accelerated.
  • TTCD deterministically evolves token embeddings from Gaussian noise at time 0 to clean token embeddings at time 1 in continuous space.Continuous-space evolution avoids the factorization problem in high-speedup settings.
  • Per-token times let tokens move from noise to data at different rates and alter how each token affects the denoising of other tokens.TTCD also maintains a nominal global time that advances from 0 to 1 in equal steps.
  • TTCD assigns rank variables to canvas tokens, with higher-ranked tokens denoised earlier.Ranks are assigned from per-token entropy after one forward pass.

2 Related Work

Prior language-generation work has developed continuous-space diffusion objectives, alternate simplex-space noising processes, and Fisher-Rao geometry. Other studies motivate time warping and propose hybrid approaches that reduce discrete sampling information loss without becoming purely continuous.

  • Continuous-space diffusion: Continuous-space diffusion work ranges from classifier-based controllable generation to embedding-space mean-squared-error objectives corresponding to the true ELBO loss.Later work replaces mean-squared loss with cross-entropy and parameterizes denoising as a convex combination of input embeddings relative to the output token distribution.
  • Alternate noising and geometry: Other approaches noise in simplex space with a Dirichlet distribution, while Fisher-Rao metrics turn probability simplexes into Riemannian manifolds for improved ELBO loss.The passage identifies DFM and DDSM as simplex-space methods and cites later work using Fisher-Rao geometry.
  • Time warping: Prior empirical work argues that time warping is necessary, making task difficulty increase linearly under metrics such as decoding error rate, validation loss, or correct-token rank.The cited time-warping methods target linearized difficulty in a chosen metric of interest.
  • Hybrid models: Hybrid models including CCDD, CADD, and CANDI seek to reduce information loss during discrete sampling but still sample tokens and are not purely continuous.These methods combine modeling approaches rather than eliminating token sampling altogether.

3 Method

TTCD extends flow matching by assigning each token its own time while refining continuous embeddings from Gaussian noise toward data. Its design ensures rank-ordered, monotonic denoising and conditions predictions on token times, enabling cleaner tokens to guide noisier positions.

  • Continuous-space formulation: TTCD starts from independent Gaussian vectors, iteratively refines them in continuous space, and maps the final vectors to a token string matching the training distribution.The target data string is represented through token embeddings before the final mapping.
  • Per-token times: Per-token times depend on a global time and independently sampled token ranks, with ranks fixed throughout each sample’s noising and denoising process.The method uses one global time tg, local times t1,...,tL, and ranks ul sampled iid from Unif(0,1).
  • Time-function design: The time function maps global noise to global data, increases monotonically with global time and rank, and keeps higher-rank tokens ahead during denoising.The boundary conditions are F(0,u)=0 and F(1,u)=1; monotonicity in u preserves rank ordering.
  • Time-function design: TTCD chooses F so that uniformly ranked local times follow Beta(1/(1−tg), 1/tg), using the Beta quantile function with appropriate boundary conditions.This construction also satisfies the stated design requirements A–D.
  • Training and denoising: The model predicts the original token string from intermediate embeddings and token times, conditioning on times to distinguish clean tokens from noisy ones.For prompt-conditioned generation, prompt-token times are fixed at 1, so prompt embeddings remain unchanged.

4 Experiments

Experiments evaluate TTCD on Sudoku, OpenWebText language generation, and QM9 molecule generation. Across these tasks, TTCD addresses parallel-sampling errors, supports conditional generation through per-token times, and improves generation quality or validity relative to discrete and continuous baselines.

  • Sudoku: TTCD combines differentiated token treatment with avoidance of the factorization problem in Sudoku generation.Per-token times distinguish known and unknown cells, while delaying commitment until the final step avoids independently sampling output tokens.
  • Sudoku: TTCD is competitive with entropy-based discrete unmasking for correctly solving Sudoku boards at low generation steps.The evaluation considers 9×9 puzzles and generation budgets from two to sixteen steps.
  • OpenWebText language generation: The 160M-parameter TTCD model is trained on OpenWebText for 1M steps and evaluated in unconditional and prefix-conditioned modes.Unconditional generation produces a full 1024-token canvas, while prefix-conditioned generation predicts 32 or 128 tokens from a clean prefix.
  • OpenWebText language generation: TTCD generation is less prone to repeating tokens, and TTCD with shortcut achieves the best prefix-conditioned generation among the evaluated diffusion models.The conditional evaluation uses a 32-token canvas at two steps (16x speedup) and four steps (8x speedup); clean-prefix token-time is fixed at one.
  • Molecule generation: Continuous-space diffusion generates more valid QM9 molecules than discrete baselines, while token-time further improves the validity frontier.Classifier-free guidance levels range from one to five, and the frontier plots ring counts against novel molecules among 1024 generations.

5 Conclusion, Limitations and Future work

TTCD is a continuous-state diffusion language model whose key innovation is token-specific time alongside global sequence time. This design improves robustness to inference-time parameters, enables prefix-conditioned generation, and outperforms the best known discrete method at high speed settings.

  • Conclusion: TTCD introduces token-time alongside global sequence-level time in a continuous-state diffusion language model.The paper identifies this separation of temporal scales as TTCD’s key differentiator.
  • Conclusion: Token-time makes TTCD significantly more robust to inference-time parameters and naturally supports prefix-conditioned generation.These properties are presented as direct benefits of including token-time.
  • Conclusion: At high speed settings, TTCD outperforms the best known discrete method on both unconditional and prefix-conditioned generation.The comparison covers both generation settings named in the conclusion.

A Theoretical Results … B.1 Shortcut Details

The theoretical appendix verifies the Beta-quantile construction’s boundary, monotonicity, stochastic-order, and sampling properties. The shortcut appendix initializes the distilled model from TTCD and trains it with multi-time conditioning and a two-step denoising target.

  • A.1 Proof of Lemma 1: F(0, u) = 0 and F(1, u) = 1 follow from the assumed boundary conventions.
  • A.1 Proof of Lemma 1: As tg approaches 0 from above, both the Beta mean and variance approach 0, so every quantile approaches 0; the tg approaches 1 limit is symmetric.The passage gives µ(tg) → 0+ and σ2(tg) → 0+ as tg → 0+ and concludes Qtg(u) → 0 for u ∈ (0, 1).
  • A.1 Proof of Lemma 1: For u ∼ Unif(0, 1), inverse transformation gives Qtg(u) ∼ Beta(α, β), and Eu[F(tg, u)] equals the Beta mean tg.
  • A.1 Proof of Lemma 1: The Beta family has a strict monotone likelihood ratio in tg, implying strict stochastic dominance.This follows because α is strictly increasing and β is strictly decreasing in tg, making the likelihood ratio strictly increasing in x.
  • B.1 Shortcut Details: The shortcut training algorithm uses the same inference procedure as standard inference, adding token-time conditioning to gϕ.The algorithm is presented as TTCD distillation using shortcut.
  • B.1 Shortcut Details: gϕ and pθ share parameters, while gϕ concatenates three token times for training stability.The algorithm initializes (gϕ, eϕ) ← (pθ, eθ).
  • B.1 Shortcut Details: Training samples tstart and tend, derives midpoint times, noises z1 to tstart, and applies gϕ for the denoising update to zmid.The midpoint times are tmid = (tstart + tend)/2, tmid,end = (tmid + tend)/2, and tstart,mid = (tstart + tmid)/2.
  • B.1 Shortcut Details: The shortcut target uses two-step unrolling from zmid and applies a consistency loss with stop-gradient on the target.The target logits are conditioned on (tmid, tmid,end, tend), while token times represent ranks and global time.

B.2 Sudoku Experimental Details

The global-only time-continuous Sudoku model uses decoding error rate as its time-warping function, following [10], with a vocabulary size of 10 for training and inference.

  • Global-only time continuous time warping: The global-only Sudoku model uses decoding error rate as its time-warping function, following [10].This warping function is used during training and inference.
  • Global-only time continuous time warping: The warping design uses a vocab size of 10.Figure 6 depicts the function used for the global-only time-continuous Sudoku model’s training and inference.

C Additional Quantitative Results · C.1 Unconditional Generation · C.2 Conditional Generation

The appendix reports additional quantitative results for unconditional and conditional generation. It varies sampling parameters for unconditional generation and evaluates conditional generation when generating either the last 32 or 128 tokens.

  • C.1 Unconditional Generation: Quantitative results for the non-distilled model are presented in Table 2.The table covers the setup described in Sec. 4.2.
  • C.1 Unconditional Generation: Quantitative results for the distilled model are presented in Table 3.The appendix reports these figures alongside the non-distilled results.
  • C.1 Unconditional Generation: Unconditional experiments vary sampling temperature and initial noise norm as sampling parameters.These parameters are plotted in Fig. 7 under the setup of Sec. 4.2.
  • C.1 Unconditional Generation: TTCD appears more stable than FLM as sampling temperature and initial noise norm vary.Figure 7 evaluates how these parameters affect generation entropy.
  • C.2 Conditional Generation: Conditional generation evaluates producing either the last 32 tokens or the last 128 tokens.Both generation settings receive figures and tables.
  • C.2 Conditional Generation: Conditional-generation figures appear in Fig. 8, while the corresponding tables are Tables 4 and 5.The appendix directs readers to these figures and tables for both settings.

C.3 Guidance Results

This section presents guidance results through generative perplexity and entropy, including distilled-model and prefix-conditioned evaluations. It also examines canvas lengths of 32 and 128 tokens and reports a QED mean figure.

  • Generative evaluation: Table 2 reports generative PPL and entropy across models and sampling parameters.
  • Generative evaluation: Table 3 reports generative PPL and entropy for distilled models.
  • Prefix-conditioned evaluation: Figure 8 presents prefix-conditioned evaluation results for canvas lengths of 32 and 128 tokens.
  • Prefix-conditioned evaluation: Prefix-conditioned evaluation reports generative PPL and entropy for a 32-token canvas.
  • Prefix-conditioned evaluation: Prefix-conditioned evaluation reports generative PPL and entropy for a 128-token canvas.
  • QED evaluation: Figure 9 presents a QED mean figure.

D Qualitative Results · D.1 Unconditional Generation

The qualitative results show TTCD generating unconditional text samples across 2-, 8-, and 32-step settings, including shortcut variants. The displayed samples contain partially coherent text alongside severe corruption and repeated-symbol artifacts.

  • D.1 Unconditional Generation: 8-step TTCD generation produces an unconditional text sample, as shown in Figure 10.The sample includes the fragment “points from the Epicberg route.”
  • D.1 Unconditional Generation: The 8-step sample contains malformed words and abrupt transitions across an apparent narrative passage.Examples include “A former WOL aschehed in pain” and a sentence fragment ending with “and”.
  • D.1 Unconditional Generation: The longer displayed sample contains corrupted wording, inconsistent names, and syntactically damaged descriptions of a trial and military personnel.The passage includes fragments about alleged assault, a not-guilty plea, and an appeal, but does not remain consistently grammatical.
  • D.1 Unconditional Generation: One displayed output degenerates into repeated arrow-like symbols rather than readable text.The excerpt consists of repeated “,→” sequences.
  • D.1 Unconditional Generation: 32-step TTCD generation is also presented as an unconditional generated sample in Figure 11.The passage identifies the setting as 32-step generation without providing a quantitative quality assessment.
  • D.1 Unconditional Generation: The qualitative results include a TTCD-with-Shortcut sample generated in 2 steps.Figure 12 documents this accelerated setting, but the supplied passage provides no textual sample or quality metric.
  • D.1 Unconditional Generation: A second TTCD-with-Shortcut example is shown for 8-step generation.Figure 13 records the setting, while the supplied passage does not state a comparative or quantitative outcome.

D.2 Conditional Generation

This section presents prefix-conditioned generation samples comparing TTCD with Shortcut against Duo with DCD using four generation steps. The examples span canvases of 32 and 128 tokens.

  • D.2 Conditional Generation: Figure 14 compares prefix-conditioned samples from TTCD with Shortcut and Duo with DCD after 4-step generation on a 32-token canvas.The displayed TTCD continuation begins with “Freedom” and discusses marijuana legalization before transitioning to an unrelated e-book reader article.
  • D.2 Conditional Generation: The TTCD sample continues a prefix about freedom into a discussion of legalizing, taxing, and regulating marijuana consumption.The passage states that freedom includes acting without interfering with a neighbor’s ability to do the same.
  • D.2 Conditional Generation: Figure 15 extends the same prefix-conditioned comparison between TTCD with Shortcut and Duo with DCD to 4-step generation on a 128-token canvas.The supplied passage identifies the figure and its generation setting but does not provide the sample text.

E Compute Requirements · NeurIPS Paper Checklist

The paper reports substantial GPU requirements for OpenWebText, Sudoku, and QM9 experiments. Its NeurIPS checklist affirms accurate claims, discussed limitations, a proved lemma, open-source data, and no LLM use, while answering no on open code access.

  • E Compute Requirements: OpenWebText training took 2 days on 32 NVIDIA H100 GPUs.All experiments used NVIDIA H100 GPUs for OpenWebText.
  • E Compute Requirements: Sudoku and QM9 training runs took around 10 hours each on a single NVIDIA A40 GPU.
  • NeurIPS Paper Checklist: The checklist states that abstract claims accurately represent the paper’s contributions and scope.It also requires claims to match theoretical and experimental results and reflect generalization limits.
  • NeurIPS Paper Checklist: The paper discusses its limitations in Section 5.The checklist asks authors to identify strong assumptions and robustness to their violations.
  • NeurIPS Paper Checklist: The paper contains one lemma and proves it in Section A.1.The checklist requires theoretical assumptions, proofs, numbering, and cross-references to be stated clearly.
  • NeurIPS Paper Checklist: The dataset is open-source, but the checklist answer for open access to data and code is [No].The authors state that an open-source repository is used for experimentation and can be used easily.
  • NeurIPS Paper Checklist: The checklist reports no such risks, credits open-source assets with appropriate citations, and introduces no new assets or crowdsourced research.
  • NeurIPS Paper Checklist: LLMs are not used as important, original, or non-standard components of the research.
Loading 2607.14106v1…