PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seunghoon Hong, Nicholas M. Boffi, Jinwoo Kim

arXiv:2602.16813v3cs.CLcs.AI

TL;DR

Discrete diffusion language models often lose quality in the few-step regime, limiting practical speedups. This paper learns continuous flows over one-hot tokens and their flow maps; FLM matches state-of-the-art discrete diffusion quality, while FMLM’s one-step generation nears recent few-step models’ 8-step quality.

  • Problem

    Discrete diffusion language models typically lose generation quality rapidly in the few-step regime, limiting the practical speedup from reducing sampling steps.

    Source: §1 · p. 1

  • Method

    The approach learns continuous flows over one-hot token embeddings and directly learns their unique flow maps for efficient few-step inference.

    Source: §1 · p. 2

  • Results

    On LM1B and OWT, FLM matches state-of-the-art discrete diffusion quality, while FMLM’s one-step generation nears recent few-step LMs’ 8-step quality.

    Source: §1 · p. 2

  • Takeaways & Limitations

    These findings challenge the belief that discrete noising is necessary for language generation over discrete modalities.

    Source: §1 · p. 2

    One-hot representations incur around 30% higher training time and memory costs than embedding diffusion methods that update only relevant embedding vectors.

    Source: §6 · p. 19

Abstract

from arXiv · show

Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

1 Introduction

The paper introduces continuous flow language models over one-hot token embeddings and learns a unique flow map for efficient few-step generation. It reports that FLM matches state-of-the-art discrete diffusion quality, while one-step FMLM approaches recent methods’ 8-step quality.

1 Introduction

  • Discrete diffusion quality typically degrades rapidly with few sampling steps because factorized reverse transitions neglect token correlations, so accurate generation requires many steps [7] [10] [11].This limits the speed benefit of parallel full-sequence sampling.Source: §1 · pp. 1–2
  • On LM1B and OWT, FLM matches state-of-the-art discrete diffusion quality, while one-step FMLM nears recent few-step models’ 8-step quality.Figure 1 states that FMLM matches distilled discrete diffusion models’ 8-step performance in one step; the authors also highlight inference-time steering and scaling via continuous guidance.Source: Figure 1 · p. 1, §1 · p. 2
  • Unlike discrete methods, continuous diffusion avoids that factorization and supports deterministic parallel generation, while flow-map distillation can transport noise to data in as few as one function evaluation [12] [13] [14] [16] [17] [18].The paper frames this as a challenge to the belief that continuous diffusion language models underperform discrete counterparts [19] [20].Source: §1 · p. 2
  • Continuous flows over one-hot token embeddings define a unique, directly learnable flow map; the authors prove discrete diffusion cannot provide this structure because its sequence state space is combinatorial.The flow map directly transports states between timepoints for efficient few-step inference.Source: §1 · p. 2
  • Simplex-valued reparameterizations enable cross-entropy objectives for the flow and flow map, which outperform square-error counterparts; a time reparameterization resolves prior continuous methods’ training instabilities.The authors also ablate key design decisions.Source: §1 · p. 2

2 Background

Language modeling seeks to efficiently sample sequences from the data distribution; autoregressive models generate serially, while discrete diffusion generates multiple tokens in parallel but struggles when sampling steps are few.

2 Background

  • Language modeling estimates the distribution over token sequences, and autoregressive models generate each token from its prefix, serializing inference and limiting efficiency [25] [26].The goal is to estimate p(y) over sequences of length L so that fresh samples can be drawn efficiently.Source: §2 · p. 3
  • Discrete diffusion aims to avoid serialization by denoising multiple tokens in parallel through learned transition densities and ancestral sampling over a temporal grid [27] [28] [29] [30] [28] [31] [27] [32].Reducing the sampling-step count substantially below sequence length L can yield speedups.Source: §2 · p. 3
  • Discrete diffusion typically fails in few-step sampling because tractable factorization assumes conditional independence that holds only infinitesimally, neglecting token correlations [7] [33] [10] [11].The paper addresses this limitation with a continuous flow formulation that learns a deterministic transport map without the factorized approximation, enabling one-step language modeling [14].Source: §2 · pp. 3–4

3 Theoretical Framework

The framework models losslessly embedded one-hot language with continuous flows and learns a flow map for few-step generation. A two-time denoiser puts flow-map learning on the simplex, while the framework contrasts tractable continuous sample-level maps with a limitation of discrete transport.

3.1 A continuous representation of language

  • The language representation is tokenwise one-hot with an argmax decoder, giving a lossless encoding that needs no regularization and operates in unconstrained Euclidean space.The paper describes this approach as simpler and more effective in practice than imposing a simplex projection on the diffusion process.Source: §3.1 · p. 4

3.2 Flow language models

  • Velocity prediction can underfit because its noised targets inherit full-rank Gaussian noise in a vocabulary-sized space, so FLM instead learns the clean-data posterior denoiser.The denoiser’s one-hot targets are structured and low-entropy; tokenwise softmax and cross-entropy constrain predictions to valid distributions on the simplex.Source: §3.2 · p. 5
  • Continuous models use the learned token posterior to infer the exact velocity, enabling few-step flow-map distillation; discrete diffusion uses it for ancestral sampling and faces factorization errors.The discrete approach requires the entire joint probability density for ancestral sampling.Source: §3.2 · p. 6

3.3 Flow map language models

  • Flow maps transport between arbitrary timepoints, enabling accurate sampling on any temporal grid, including one evaluation, unlike large-step numerical integration of the flow.The authors focus on semigroup-condition distillation for simplicity; differential alternatives require Jacobian-vector products and are left for future empirical work.Source: §3.3 · pp. 6–7

The two-time denoiser.

  • The two-time denoiser lies on the simplex, recovers the flow map exactly, matches the standard denoiser on the diagonal, and satisfies a semigroup condition.Its simplex property permits tokenwise softmax parameterization and KL-based learning of the semigroup condition.Source: The two-time denoiser · pp. 7–8
  • The two-time denoiser’s KL semigroup objective supports distillation from a pretrained denoiser, while replacing its diagonal target with one-hot data gives direct training without a teacher.The KL objective has a zero minimum and measures reconstruction quality; the direct-training diagonal term recovers the single-time denoiser’s cross-entropy loss.Source: The two-time denoiser · p. 8
  • For finite state spaces, some target distributions cannot be obtained by pushing a given distribution through any deterministic map, unlike tractable continuous sample-level flow maps.The proposition establishes this limitation for any probability distribution on a finite set; the paper contrasts it with continuous flows’ sample-level maps.Source: The two-time denoiser · p. 8

4 Algorithmic Aspects

The section sets out practical choices for training, distilling, and sampling FLM/FMLM, including reparameterized time and boundary sampling for few-step generation. It also describes inference-time guidance methods enabled by continuous flows and flow maps.

Time reparameterization

  • Time reparameterization allocates training and inference steps to the narrow interval where token identities are resolved, rather than using uniform time in t.The schedule makes equal steps in reparameterized time correspond to equal reductions in decoding error and supports scaling FLM to vocabularies of about 50,000 tokens.Source: §4 · p. 10, Figure 9 · p. 10

Distillation

  • FMLM distillation trains a two-time denoiser against a frozen FLM teacher, using diagonal and off-diagonal losses with continuously sampled time pairs.The midpoint partitions each interval for the semigroup condition; unlike shortcut models [47], continuous sampling trains across timescales, and generation uses the reparameterized-time grid.Source: §4 · p. 10

Boundary sampling

  • Boundary sampling addresses sparse training near t = 0 by explicitly sampling (s, t) = (0, 1), improving flow-map learning for one- or two-step generation.The paper reports p = 1/32 as working well in practice.Source: §4 · p. 11

Inference-time guidance

  • Continuous flows support inference-time guidance directly on velocity or denoiser predictions, whereas discrete diffusion guidance extrapolates in logit space and can amplify factorization artifacts.The section demonstrates autoguidance and reward-guided generation as proof-of-concept techniques.Source: §4 · p. 11
  • Autoguidance [49] improves unconditional sample quality by extrapolating the learned velocity field away from a weaker model.The guidance strength η > 1 controls the extrapolation, which for continuous flows occurs in Euclidean velocity space.Source: §4 · p. 11
  • Flow Map Reward Guidance [50] alternates flow-map steps with reward-gradient steps, evaluating reward at the endpoint so the reward model needs training only on clean data.The flow map provides a differentiable look-ahead from an intermediate state; this avoids training a discrete-diffusion classifier across the full noising trajectory and also supports terminal-reward rollouts for reinforcement learning finetuning.Source: §4 · p. 11

5 Experiments

Experiments on LM1B and OWT evaluate flow and flow-map language models using generative perplexity and sample entropy, alongside inference-time guidance, design ablations, and Sudoku generation and solving.

5.1 Flow language model

  • At 1024 sampling steps, FLM beats all baselines on LM1B sample quality while preserving entropy, and has the best OWT perplexity with entropy within ±0.1 of data entropy.Across 8–1024 steps, FLM is competitive, but distillation into FMLM is needed for very few-step performance.Source: Table 10 · p. 12, Figure 11 · p. 12, §5.1 · pp. 13–14

5.2 Flow map language model

  • 119.34 one-step Gen. PPL on LM1B matches distilled baselines at 8–16 steps, while 168.30 on OWT is comparable to 4–8 steps with entropy 5.17.FMLM remains stable across step counts; its one-step samples are coherent and grammatical, whereas discrete baselines produce incoherent or repetitive text.Source: Figure 12 · p. 13, Table 13 · p. 13, Figure 14 · p. 14

5.3 Inference-time guidance

  • 51.62 Gen. PPL at η = 50 improves on FLM’s unguided 96.91 at 1024 steps, with entropy 4.00; Duo and MDLM collapse at η ≥10.FLM remains stable across guidance scales up to 100, unlike the discrete baselines; the paper attributes their collapse to multiplicative logit-space guidance amplifying token-correlation errors.Source: §5.3 · p. 15, Figure 16 · p. 16
  • FMLM produces high-reward outputs aligned with target properties while preserving sample quality even at 2 steps.The reward-guidance evaluation covers topic, grammaticality, sentiment, and safety; the table caption reports higher reward scores and lower generative perplexity than discrete baselines across step counts.Source: Table 17 · p. 16, §5.3 · p. 17

5.4 Ablation study

  • Denoiser prediction with softmax and cross-entropy performs best, while velocity prediction fails to converge; decoding-error reparameterization and one-hot encodings outperform the tested alternatives.The unconstrained Euclidean interpolant beats Riemannian and simplex diffusion; simplex diffusion has entropy 3.76, which the authors hypothesize reflects low diversity from uniform initialization in high dimensions.Source: §5.4 · pp. 17–18

5.5 Modeling logical structures

  • 9% of FMLM’s one-step unconditional Sudoku generations are valid, while one-step discrete baselines approach zero validity; its 1024-step validity is near-perfect.FMLM reaches near 40% one-pass accuracy on held-out puzzles and 16% on Sudoku-Extreme-Full; rejection sampling gives approximately 7.6× speedup over 81-step autoregressive generation.Source: Table 22 · p. 18, §5.5 · p. 19

5.4 Ablation study

  • The two-time denoiser with cross-entropy outperforms average-velocity squared-loss training, and distillation from a pretrained FLM beats direct self-distillation in the tested compute ranges.The semigroup objective and midpoint triplet sampling outperform the tested alternatives; including the boundary pair (0, 1) with probability 1/32 improves both perplexity and entropy.Source: §5.4 · p. 18

6 Conclusion · A Related work · B Background on flow maps

The paper presents continuous flows over one-hot token embeddings as a language-modeling approach that matches discrete diffusion in many-step generation and improves few-step generation, including one-step sampling. It grounds flow-map learning and inference in a two-time denoiser, guidance capabilities, and flow-map characterizations.

6 Conclusion

  • FLM matches state-of-the-art discrete diffusion in the many-step regime, while FMLM substantially outperforms distilled discrete methods in the few-step regime, including one-step generation.The paper reports that continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed.Source: §6 · p. 19
  • The two-time denoiser places the flow map on the simplex for cross-entropy training, while differentiable look-ahead enables autoguidance and reward-guided generation.The paper also identifies continuous-model tools such as guidance, editing, and inversion as opportunities for language generation, and suggests FMLM may reduce rollout computation and memory for reinforcement-learning finetuning.Source: §6 · p. 19
  • The one-hot representation requires full |V| × d embedding-matrix evaluation and backpropagation at each training step, costing around 30% more time and memory than embedding diffusion.Sparse gradients or structured representations are proposed as possible ways to address this limitation.Source: §6 · p. 19

A Related work

  • Discrete diffusion reverses masking or randomized-subword noising, but factorized reverse transitions introduce irreducible error that hinders few-step generation [7] [11]; the authors find purely continuous methods may suffice.Some prior work combines discrete and continuous diffusion [20] [73].Source: A Related work · p. 27
  • Continuous language diffusion commonly uses learned or frozen embeddings [12] [34] [74] [35] [36], whereas this work uses unconstrained Euclidean one-hot flows; dequantization can yield full-rank denoising targets in large vocabularies [77].The authors identify CDCD [13] as most related, noting its learned embeddings and loss-based time reparameterization requiring online estimation.Source: A Related work · p. 27
  • Continuous few-step methods use Eulerian, Lagrangian, or semigroup flow maps [43] [44] [45] [47] [80]; the authors adopt semigroup methods for simplicity, while discrete consistency distillation retains factorization error [7] [19] [59].A concurrent work applies flow-map distillation to continuous language interpolants [81].Source: A Related work · p. 27

B Background on flow maps

  • A flow map is the unique arbitrary-time jump operator along probability-flow trajectories and satisfies Lagrangian, Eulerian, and semigroup characterizations.These characterize forward trajectory evolution, dependence on the starting time, and replacement of successive jumps by a direct jump, respectively; they underpin Lagrangian self-distillation and terminal velocity matching [45], consistency models and MeanFlow [43], and progressive distillation [46] and shortcut models [47].Source: B Background on flow maps · pp. 27–28
  • The flow map encodes the velocity field on its diagonal, and its application from time 0 to 1 enables one-step sampling without iterative integration.A learned two-time function can represent average trajectory velocity off the diagonal; composing maps over a time grid trades additional computation for refinement.Source: B Background on flow maps · p. 28

B.1 Direct training and distillation of flow maps · C Theoretical details · C.1 Proofs from the main text

Flow maps can be learned by distilling a pre-trained velocity field or by direct self-distillation. The appendix characterizes both approaches through losses and describes the tangent condition as a basis for direct training.

B.1 Direct training and distillation of flow maps

  • Flow-map learning has two routes: distillation from a pre-trained velocity model and direct self-distillation without a pre-trained teacher.The appendix summarizes both approaches.Source: §B.1 · p. 28
  • For a pre-trained velocity field, the flow map is the unique critical point of Lagrangian, Eulerian, and progressive map distillation losses.These objectives are derived from characterizations in Proposition B.2.Source: §B.1 · pp. 28–29
  • Distillation converts a pre-trained velocity field into a flow map and is typically faster and less compute-intensive than self-distillation, especially when large-scale pre-trained models are available.The appendix also notes that training flow maps from scratch remains useful.Source: §B.1 · p. 29
  • Direct flow-map training lacks an obvious learning target; the tangent condition (32) provides one by enabling systematic flow-matching training of the diagonal velocity.Combining this insight with the distillation objectives yields a single-phase training approach.Source: §B.1 · p. 29
  • For self-distillation, the flow map is the unique critical point of Lagrangian, Eulerian, and progressive self-distillation losses.The losses are named LSD, ESD, and PSD, respectively.Source: §B.1 · p. 29
  • LSD has been scaled as Terminal Velocity Matching [45], ESD is equivalent to Improved MeanFlow [44], and PSD is a continuous-time limit of shortcut models [47].Proofs are referred to Boffi et al. [17].Source: §B.1 · p. 29

C.1.1 Proof of Lemma 3.1

The proof establishes that the optimal denoiser at each token position is the token-level posterior, so its output lies on the vocabulary simplex.

C.1.1 Proof of Lemma 3.1

  • The optimal denoiser is the token-level posterior and therefore lies on the vocabulary simplex.The proof identifies the denoiser output with posterior probabilities over the vocabulary.Source: §C.1.1 · p. 30
  • At token position l, the proof uses that the clean token x_1^l is one-hot, taking values among the vocabulary basis vectors.The basis vector e_i denotes the one-hot encoding of the i-th subword.Source: §C.1.1 · p. 30
  • Expanding the conditional expectation over those one-hot values yields the vector of posterior probabilities over the vocabulary.This expansion gives the token-level posterior identified as the optimal denoiser.Source: §C.1.1 · p. 30

C.1.2 Proof of Proposition 3.2

The proof establishes that cross-entropy uniquely identifies the optimal denoiser and that its excess risk controls the learned sampler’s Wasserstein error under regularity assumptions. It connects these results through simplex-based denoiser error bounds and a truncated-flow argument.

Proof of Proposition 3.2

  • Cross-entropy decomposes into irreducible conditional entropy and a KL gap, so its unique minimizer predicts the true conditional token distribution.Nonnegativity of KL divergence, with equality only when the distributions match, establishes uniqueness.Source: §C.1.2 · pp. 30–31
  • Under the stated regularity assumptions, cross-entropy excess risk controls the Wasserstein distance between the learned sampler’s distribution and the target distribution.The proof truncates the sampler at 1 − ξ; the constant in the bound depends on ξ and the model’s Lipschitz constant.Source: §C.1.2 · pp. 30–32
  • The flow bound assumes Lipschitz velocities, bounded second moments, and simplex-interior denoiser outputs; it couples flows from the same initial sample and applies Gronwall’s inequality.A terminal remainder handles the interval [1 − ξ, 1] using the second-moment bounds.Source: §C.1.2 · pp. 31–33
  • Pinsker’s inequality uses the simplex structure to bound L2 denoiser error by excess risk, while the denoiser-to-velocity conversion amplifies error by (1 − t)^−1.The proof therefore integrates only through time 1 − ξ to control velocity error.Source: §C.1.2 · pp. 31–32

C.1.3 Proof of Proposition 3.5 … C.6 Decoding error rate and entropic time reparameterizations

The appendix motivates continuous-flow maps by showing that discrete diffusion cannot deterministically transport every noise distribution to every target, and develops simplex-valued two-time denoisers and their learning objectives. It also connects decoding-error-based time reparameterization to entropic time using Fano’s inequality, noting the bound’s uniform-token assumption and how a Zipf-like distribution changes it.

C.1.3 Proof of Proposition 3.5

  • For any finite-set noise distribution, some target distribution is unreachable by a deterministic map, unlike continuous flows, which admit a sample-level flow map.A deterministic pushforward assigns each output either zero mass or at least the smallest positive input mass; choosing a target atom with half that mass proves the claim.Source: §C.1.3 · p. 34

C.2 Flow maps on the simplex

  • The endpoint denoiser is the posterior mean of clean data, while the two-time denoiser reparameterizes mean flow as a simplex-valued clean-data predictor.The endpoint denoiser corresponds to one Euler step of size 1−t; the flow map instead gives the exact ODE solution.Source: §C.2 · pp. 34–35

C.3 The two-time denoiser.

  • The two-time denoiser recovers the flow map exactly, lies on the simplex, matches the standard denoiser on the diagonal, and satisfies a semigroup condition.Its simplex property follows by expressing it as a weighted average of single-time denoisers along the flow trajectory.Source: §C.3 · p. 35

C.4 Characterizing the two-time denoiser

  • Flow-map consistency translates into Lagrangian, Eulerian, and semigroup conditions on the two-time denoiser, with the semigroup relation a convex combination when s ≤ u ≤ t.At u = (s + t)/2, the weights are γ = (1−t)/(2−s−t) and 1−γ = (1−s)/(2−s−t).Source: §C.4 · pp. 36–37

C.5 Learning the two-time denoiser

  • Distillation and self-distillation learn the unique two-time denoiser by enforcing one flow-map condition off-diagonal and anchoring the diagonal to the denoiser.Semigroup teachers lie on the simplex and use KL; Lagrangian and Eulerian teachers may transiently leave it, so their losses use cross-entropy with a softmax student.Source: §C.5 · pp. 38–40

C.6 Decoding error rate and entropic time reparameterizations

  • Entropic time reparameterization [13] [48] allocates equal uncertainty resolution across time by linearizing clean-data conditional entropy; Dieleman et al. [13] estimate it online from denoiser training loss.Source: §C.6 · p. 41
  • Under a uniform token-distribution assumption, Fano’s inequality relates decoding error to conditional entropy to establish an approximate asymptotic bound with a vanishing margin.The authors note that the assumption is strong; for a Zipf-like distribution, the bound instead takes the form σ + O(1 / log |V|) ≥ 1 − cPe for some c > 1.Source: §C.6 · p. 41
  • The connection suggests entropic-time underperformance may stem from training-loss-based approximation rather than entropy linearization.Online-estimated entropic time [13] underperformed the proposed time reparameterization in Section 5.4.Source: §C.6 · p. 41

D Two-stage flow map distillation with squared loss

The section presents a two-stage squared-loss route to FMLM: a correction model improves Euler-based flow-map jumps, then a single-model student compresses the resulting two-model teacher. First-stage results are comparable to cross-entropy distillation on both datasets, and the LM1B student retains teacher performance.

Motivation

  • The usual squared-loss distillation does not use one-hot geometry and is outperformed by cross-entropy, but the two-stage design offers another route that can stabilize squared-loss distillation.Source: D Two-stage flow map distillation with… · p. 42

First stage

Results

Second stage

  • The second stage regresses a single-model flow map onto the frozen two-model teacher; one-pass targets and a nonnegative, strongly convex loss with a unique minimizer support stable optimization.Source: D Two-stage flow map distillation with… · p. 43

Time reparameterization and other details

Results

E Implementation details

The implementation uses a cached spline lookup table for time reparameterization, shared training settings for FLM and FMLM with boundary sampling for few-step distillation, and specified samplers and checkpoint sources for model comparisons.

Time reparameterization

  • A cached 1,000-point lookup table with cubic-spline interpolation provides forward and inverse time mappings for O(1) simulation-time sampling without extra training overhead.The table approximates the schedule CDF using Gauss-Hermite quadrature; the authors report negligible error at this resolution.Source: E Implementation details · p. 44

Training details

  • FLM and FMLM use batch size 512 and learning rate 3 × 10−4; FLM trains for 1M steps, while FMLM distillation runs for 100k steps.FMLM splits batches equally between flow matching (s = t) and flow map distillation (s ≠ t), and samples (s, t) = (0, 1) with probability 1/32 to train few-step generation.Source: E Implementation details · p. 44

Sampling details

  • FLM uses an Euler solver on LM1B and OWT, while FMLM uses γ-sampling with optimal γ values [92] on both datasets.The γ-sampling algorithm is from Kim et al. [91].Source: E Implementation details · p. 44

Many-step baselines

  • For LM1B, Duo [19], MDLM [29], and CANDI [20] were trained from scratch, while RDLM [58] used its official checkpoint; OWT comparisons used official checkpoints for CANDI, Duo, and MDLM.RDLM was not compared on OWT because no official checkpoint was available and reproduction resources were limited; discrete baselines used ancestral sampling at temperature 1.0, except RDLM, which used its paper's SDE sampler.Source: E Implementation details · p. 44

Few-step baselines

  • LM1B few-step baselines used SDTT [10] on MDLM [29] and DCD [19] on Duo, each with 8 distillation rounds of 10k steps; OWT used official distilled checkpoints.Di4C [59] used checkpoints with the best 32-step performance at 20k training steps for LM1B and 50k for OWT, since further training degraded performance.Source: E Implementation details · p. 45

F Supplementary evaluation results

Supplementary evaluations report that FMLM avoids mode collapse and performs Sudoku generation, while FLM improves with model scale. Ablations and cost comparisons describe how prior choice, flow-map distillation, and computation affect these models.

Checking mode collapse

  • FMLM’s one-step Self-BLEU is only slightly worse than real data, while GPT-4.1 diversity win rates of 39% and 42% indicate slightly decreased diversity, not mode collapse.The passage attributes unusually high baseline win rates to near-random token outputs and generative perplexity above 1000.Source: F Supplementary evaluation results · p. 45

Scaling behavior of FLM

  • Increasing FLM capacity from 179M to 870M parameters consistently improves generative perplexity while preserving sample entropy.The evaluation compares GPT-2 Small (179M), Medium (424M), and Large (870M) models at a constant number of training iterations; 179M models are used for the paper’s main results.Source: F Supplementary evaluation results · p. 46

Ablation on noise distribution

  • The Gaussian prior outperforms the uniform simplex prior because the latter concentrates near the uniform vector, collapsing effective training signal into a narrow time interval.For vocabulary size |V|, Dir(1) samples have per-component variance ∼1/|V|^2; Gaussian noise remains far from the data manifold, allowing decoding error to decrease gradually across the trajectory.Source: F Supplementary evaluation results · p. 46, Table 28 · p. 46

Ablation on flow map characterization

  • The semigroup flow-map distillation objective converges stably and gives better generation quality than Lagrangian distillation, while Eulerian distillation is highly unstable.Eulerian requires spatial Jacobian computation over the full L × |V| state; semigroup avoids costly JVP operations.Source: Figure 31 · p. 46, F Supplementary evaluation results · p. 47

Computational Cost

  • FLM uses roughly 15–20% more training memory than MDLM but is consistently faster, while Duo costs up to 14% more memory and 25% more time than FLM at |V| = 50k and L = 1024.FLM’s O(L|V|) memory and O(L|V|d) time per forward pass match discrete diffusion asymptotically; FLM also avoids ancestral sampling.Source: F Supplementary evaluation results · p. 47, Table 32 · p. 47

Uniqueness and novelty in Sudoku generation

  • FMLM achieves near-perfect Sudoku uniqueness and novelty at 1024 steps, and 5% validity and uniqueness at one step, outperforming few-step distilled discrete diffusion baselines.The reported validity, uniqueness, and novelty values coincide because models produce unique valid grids within 1024 generations.Source: Table 29 · p. 46, Table 30 · p. 46, F Supplementary evaluation results · p. 47

G Supplementary qualitative results

The supplementary samples compare one-step conditional generation, fixed-noise refinement across sampling steps, and guided generation toward safety and topic attributes.

G Supplementary qualitative results

Related papers

Loading 2602.16813v3…