PaperlayerView exact revision on arXiv

Source-linked AI summary

Flow Map Language Models: One-step Language Modeling via Continuous Denoising

Chanhyuk Lee, Jaehoon Yoo, Manan Agarwal, Sheel Shah, Jerry Huang, Aditi Raghunathan, Seunghoon Hong, Nicholas M. Boffi, Jinwoo Kim

arXiv:2602.16813v3cs.CLcs.AI

TL;DR

Discrete diffusion language models often lose quality when reduced to a few sampling steps, limiting their speed advantage. This paper introduces continuous flow and flow-map language models over one-hot token embeddings, showing that FMLM one-step generation substantially outperforms distilled discrete methods in the few-step regime.

  • Problem

    Discrete diffusion language models typically suffer sharp quality degradation in the few-step regime, limiting practical acceleration despite parallel sequence processing.

  • Method

    The paper learns continuous flows and their simplex-valued flow maps over one-hot token embeddings using cross-entropy objectives and flow-map distillation.

  • Results

    FMLM substantially outperforms distilled discrete methods in the few-step regime, including one-step generation, while FLM matches state-of-the-art discrete diffusion quality in many steps.

  • Takeaways & Limitations

    Continuous flow maps provide a practical route to accelerated language modeling and support inference-time guidance unavailable to discrete methods.

  • Takeaways & Limitations

    The one-hot representation incurs around 30% higher training time and memory costs than embedding diffusion methods that update only relevant embedding vectors.

Abstract

from arXiv · show

Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

1 Introduction

Discrete diffusion language models enable parallel sequence generation but lose quality rapidly with few sampling steps, while continuous flows over one-hot embeddings offer a deterministic, directly learnable flow map for efficient inference. The paper introduces FLM and distilled FMLM models that match state-of-the-art discrete diffusion quality and approach eight-step quality in one step.

  • Motivation: Discrete diffusion language models’ quality drops rapidly with few sampling steps, because factorized reverse transitions neglect token correlations in a combinatorially large sequence space.This makes many sampling steps necessary to accurately capture full sequences, reducing their practical speed advantage over autoregressive models.
  • Motivation: Continuous flows over one-hot token embeddings define a unique flow map that can be directly learned for efficient few-step inference.The paper argues that discrete diffusion cannot provide this structure because its sequence state space is combinatorially large.
  • Method: The approach reparameterizes the flow and flow map as simplex-valued objects and introduces cross-entropy objectives that respect discrete data geometry.These objectives dramatically outperform standard square-error objectives, while time reparameterization resolves training instabilities of prior continuous methods.
  • Method: The authors build a flow language model (FLM) and distill it into a flow map language model (FMLM) for direct transport between distant timepoints.This formulation enables few-step generation through a continuous flow and its distilled flow map.
  • Results: On LM1B and OWT, FLM matches state-of-the-art discrete diffusion quality, while FMLM beats recent few-step models and nears their 8-step quality at one step.The paper also highlights inference-time steering and scaling via continuous guidance as downstream advantages of FMLM.

2 Background

Language modeling estimates a distribution over token sequences for efficient sampling, but autoregressive models serialize generation while discrete diffusion models approximate full-sequence transitions and degrade in the few-step regime. The factorized diffusion approximation neglects token correlations except in the infinitesimal limit, making reduced-step sampling fundamentally problematic.

  • Language modeling: Language modeling estimates p(y) over length-L token sequences y ∈ V^L so that fresh samples can be drawn efficiently.The vocabulary V is treated as integer-indexed tokens.
  • Autoregressive models: Autoregressive models factorize the data distribution into token conditionals p(y_l|y_<l), requiring sequential sampling conditioned on preceding tokens.This serialization limits generation efficiency and motivates methods that model the full sequence simultaneously.
  • Discrete diffusion: Discrete diffusion generates multiple tokens in parallel by reversing masking or uniform-randomization noise through transition densities over a temporal grid.Generation uses ancestral sampling over 0 = t_0 < . . . < t_N = 1.
  • Discrete diffusion: Discrete diffusion typically fails catastrophically in the few-step regime because tractable factorization assumes denoised tokens are conditionally independent.The assumption holds only as t → s, so fewer sampling steps neglect token correlations and produce unnatural text; this is described as fundamental rather than solvable through model-quality improvements alone.

3 Theoretical Framework

The framework models language as a continuous flow over canonical one-hot token embeddings, using flow matching and denoising to learn the probability flow. A two-time denoiser enables simplex-respecting flow-map learning, while discrete diffusion generally lacks an analogous deterministic sample-level map.

  • Continuous representation: The model uses a lossless tokenwise one-hot representation with an argmax decoder in unconstrained Euclidean space.This avoids embedding regularization, auxiliary training, and simplex projection constraints.
  • Probability flow: Flow matching connects Gaussian noise to data through a stochastic interpolant and learns a probability-flow velocity field for numerical sampling.Training samples time uniformly and integrates the learned flow over a temporal grid.
  • Denoising: The denoiser replaces direct velocity prediction because full-rank Gaussian targets can underfit networks whose feature dimension is much smaller than the vocabulary dimension.Predicting structured, low-entropy one-hot data also yields a token-level posterior interpretation.
  • Flow maps: The flow map directly transports samples between arbitrary timepoints, overcoming the inaccuracy of large-step numerical solvers.This structure underlies exact distillation of continuous flows into few-step generators.
  • Two-time denoiser: The two-time denoiser exactly recovers the flow map while remaining on the probability simplex and satisfying diagonal and semigroup consistency properties.Its simplex-valued form supports tokenwise softmax parameterization and KL-based semigroup training.
  • Discrete flow maps: For finite discrete state spaces, some target distributions cannot be obtained by deterministic pushforwards, whereas continuous flows admit tractable sample-level flow maps.This establishes a structural limitation for deterministic discrete flow maps and an efficiency advantage for continuous flows.

4 Algorithmic Aspects

The practical FLM/FMLM design uses reparameterized time for training and inference, distills a two-time denoiser into a flow map, and supports guidance methods enabled by continuous flows. Sampling algorithms iteratively update the state using either predicted velocity or flow-map denoising transitions.

  • Algorithms: FLM sampling advances states with denoiser-predicted velocity, while FMLM sampling uses a learned two-time denoiser to transport between successive reparameterized-time points.Both procedures initialize x0 ∼N(0, I), iterate over N steps, and return g(xtN).
  • Time reparameterization: Reparameterized time allocates denoising steps uniformly, concentrates samples near the sharp decision boundary, and enables FLM scaling to |V| ≈50,000.Training samples t through τ(t) ∼U[0, 1], while inference uses tn = t(n/N).
  • Distillation: FMLM distillation trains a tokenwise-softmax two-time denoiser against a frozen pretrained FLM with the KL objective, then recovers the flow map for sampling.The same reparameterization is used for training and inference, with diagonal and off-diagonal time sampling; boundary-pair sampling uses p = 1 32.
  • Inference-time guidance: Continuous flows support inference-time guidance in Euclidean velocity space, whereas discrete diffusion extrapolates in logit space under a factorized approximation that can amplify artifacts.Autoguidance extrapolates away from a weaker model with guidance strength η > 1.
  • Reward-guided generation: The flow map enables reward-guided generation by providing differentiable look-ahead to the endpoint, so reward models need training only on clean data.Flow Map Reward Guidance alternates flow-map steps with reward-gradient steps and evaluates rewards on the near-one-hot endpoint output.

5 Experiments

Experiments on LM1B and OWT show that FLM outperforms discrete diffusion in the many-step regime, while FMLM transfers this quality to stable one-step generation. Additional results demonstrate strong conditional, guided, qualitative, and constrained-generation performance, alongside design choices favoring geometry-aware objectives and pretrained-teacher distillation.

  • Many-step generation: FLM outperforms diffusion baselines on LM1B and achieves the best generative perplexity on OWT while maintaining sample entropy close to the data.The comparison uses 1024-step sampling and evaluates generative perplexity alongside entropy to detect repetitive outputs.
  • Many-step generation: FLM is competitive from 8 to 1024 sampling steps, but distillation into FMLM is necessary for very few-step performance.The results show that continuous denoising outperforms discrete diffusion methods in the many-step regime, while FLM quality degrades at low step counts before distillation.
  • Few-step generation: 119.34: FMLM’s one-step LM1B Gen. PPL matches distilled baselines at 8–16 steps, while 168.30 on OWT is comparable to baselines at 4–8 steps.OWT one-step generation also maintains entropy of 5.17, and FMLM remains stable across all step counts.
  • Qualitative results: FMLM generates coherent one-step LM1B samples, whereas baselines produce incoherent or repetitive token sequences.The baseline failures reflect difficulty representing correlations between tokens.
  • Conditional generation: 141.66: FMLM’s conditional OWT Gen. PPL is 2.6× lower than MDLM+SDTT after finetuning on 50-token conditioning prefixes.The evaluation uses 256 prefixes with five generations per prefix and generates 50-token continuations.
  • Guidance: 51.62: autoguidance reduces FLM Gen. PPL from 96.91 at 1024 steps while maintaining entropy of 4.00, whereas Duo and MDLM collapse at η ≥10.The reported collapse thresholds are Gen. PPL > 1900 for Duo and Gen. PPL > 1750 for MDLM.
  • Design ablations: Denoiser prediction with softmax and cross-entropy is the best FLM design, while FMLM benefits from two-time cross-entropy, pretrained-FLM distillation, and the semigroup objective.The unconstrained Euclidean interpolant also outperforms Riemannian and simplex diffusion, whose entropy collapses to 3.76.

6 Conclusion · A Related work · B Background on flow maps

The paper concludes that continuous flows over one-hot embeddings enable high-quality, faster language generation, with FMLM supporting strong few-step and one-step sampling. It situates this approach within discrete and continuous diffusion research and formalizes flow maps as unique, composable solution operators underlying efficient sampling and distillation.

  • 6 Conclusion: Continuous flows over one-hot embeddings outperform discrete diffusion in both quality and speed, while FLM matches many-step baselines and FMLM improves few-step generation.FMLM also exceeds distilled discrete methods in the one-step regime.
  • 6 Conclusion: The two-time denoiser places the flow map on the simplex and enables cross-entropy training for the proposed language models.The conclusion identifies this reparameterization as central to the approach.
  • 6 Conclusion: The approach connects language generation to continuous-model tools such as guidance, editing, and inversion, while motivating larger-scale flow models and cheaper reinforcement-learning rollouts.These are presented as broader opportunities enabled by the continuous formulation and FMLM.
  • 6 Conclusion: The one-hot representation incurs around 30% higher time and memory costs than embedding diffusion because training processes the full |V | × d embedding matrix.Sparse gradients or structured representations are suggested as possible remedies.
  • A Related work: Discrete diffusion language models reverse masking or subword-randomization processes, but factorized reverse transitions introduce irreducible error that hinders few-step generation.This limitation motivates alternatives that combine or replace discrete diffusion.
  • A Related work: Continuous diffusion language models generally denoise learned or frozen embeddings, while one-hot approaches have mainly used simplex or Riemannian formulations rather than the unconstrained Euclidean setting.The paper positions its representation and geometry within this existing landscape.
  • A Related work: Few-step generative modeling uses flow maps that jump between arbitrary timepoints, with Eulerian, Lagrangian, and semigroup approaches; this work adopts semigroup methods for computational simplicity.All three approaches are described as compatible.
  • B Background on flow maps: A flow map is the unique solution operator of the probability-flow equation, supports Lagrangian, Eulerian, and semigroup characterizations, and enables one-step sampling or multi-step refinement by map composition.Given x0 ∼p0, one application x1 = X0,1(x0) produces a sample from p1; composing maps over a time grid trades compute for quality.

B.1 Direct training and distillation of flow maps · C Theoretical details · C.1 Proofs from the main text

Flow maps can be learned by distillation from a pre-trained velocity field or directly through self-distillation. Distillation offers faster, lower-compute conversion, while self-distillation uses the tangent condition to construct training targets without a teacher.

  • B.1 Direct training and distillation of flow maps: Flow maps support two training routes: distillation from a pre-trained velocity model and direct self-distillation without a pre-trained teacher.The section summarizes both approaches.
  • B.1 Direct training and distillation of flow maps: A pre-trained velocity field ˆbt can be distilled into a flow map ˆXs,t using objectives derived from Proposition B.2.The resulting flow map is characterized as the unique critical point of the distillation losses.
  • B.1 Direct training and distillation of flow maps: The three distillation objectives are Lagrangian map distillation, Eulerian map distillation, and progressive map distillation.They are abbreviated LMD, EMD, and PMD, respectively.
  • B.1 Direct training and distillation of flow maps: Distillation is typically faster and requires less compute than self-distillation, especially when large-scale pre-trained models are available.The objectives provide a mechanism for converting a pre-trained velocity field into a flow map.
  • B.1 Direct training and distillation of flow maps: Direct flow-map training addresses the absence of an obvious learning target by using the tangent condition to train the diagonal ˆvt,t via flow matching.This construction combines the tangent-condition insight with the distillation objectives.
  • B.1 Direct training and distillation of flow maps: Self-distillation uses Lagrangian, Eulerian, and progressive losses, respectively called LSD, ESD, and PSD.The flow map is stated to be the unique critical point of these losses.
  • B.1 Direct training and distillation of flow maps: LSD, ESD, and PSD correspond respectively to Terminal Velocity Matching, Improved MeanFlow, and a continuous-time limit of shortcut models.These connections are reported as relationships to prior methods.

C.1.1 Proof of Lemma 3.1

The proof shows that the optimal denoiser equals the token-level posterior. Because the conditional expectation over one-hot token encodings yields posterior probabilities, the denoiser lies on the vocabulary simplex.

  • C.1.1 Proof of Lemma 3.1: The optimal denoiser is given by the token-level posterior.
  • C.1.1 Proof of Lemma 3.1: For each token position, the proof derives the denoiser from Equation (9).
  • C.1.1 Proof of Lemma 3.1: Since the token representation is one-hot, it takes values among the vocabulary basis vectors e_1 through e_|V|.
  • C.1.1 Proof of Lemma 3.1: Expanding the conditional expectation produces the vector of posterior probabilities over the vocabulary, so the denoiser lies on the simplex Δ_|V|−1.

C.1.2 Proof of Proposition 3.2

The proof establishes that cross-entropy uniquely recovers the optimal denoiser and that its excess risk controls the learned ODE sampler’s Wasserstein error. It connects these results through simplex-based KL and L2 bounds, velocity-error control before 1−ξ, and a Gronwall flow comparison under regularity assumptions.

  • Proof of the minimizer: The optimal denoiser is the unique minimizer of the cross-entropy objective.The proof uses cross-entropy’s decomposition into conditional entropy and a nonnegative KL divergence, with equality only when the predicted distribution matches the true conditional distribution.
  • Proof of the minimizer: The cross-entropy decomposition separates irreducible conditional entropy from the learnable diagonal excess-risk KL gap.The entropy term depends only on the data distribution and cannot be reduced by the model.
  • Wasserstein bound: Lipschitz velocities, finite second moments, and simplex-interior outputs support a Gronwall comparison that converts velocity error into a Wasserstein flow bound.The terminal interval [1−ξ, 1] is handled separately through short-interval remainder bounds.
  • Wasserstein bound: Under simplex-interior outputs, Pinsker’s inequality converts excess risk into an L2 denoiser-error bound.The simplex-interior property is guaranteed in practice by a softmax output layer.
  • Wasserstein bound: The denoiser-to-velocity relation amplifies errors by (1−t)^−1, so the sampler is truncated at 1−ξ before bounding velocity error.The velocity bound applies for any ξ ∈(0, 1).

C.1.3 Proof of Proposition 3.5 · C.2 Flow maps on the simplex

The proof shows that deterministic one-step transport on a finite discrete space cannot represent every target distribution, whereas continuous flows provide sample-level flow maps. The simplex-flow analysis then relates endpoint denoisers to velocity fields and distinguishes denoising Euler steps from exact flow-map solutions.

  • C.1.3 Proof of Proposition 3.5: Proposition 3.5 states that for any distribution µ on a finite set S, some distribution ν cannot equal f#µ for any deterministic map f.This establishes an expressivity limitation for deterministic transport on finite discrete spaces.
  • C.1.3 Proof of Proposition 3.5: Every deterministic pushforward assigns each output probability as a sum of input probabilities, so positive masses are at least µmin.Choosing ν(y0) = µmin/2 constructs an unreachable target distribution.
  • C.1.3 Proof of Proposition 3.5: For discrete diffusion with S = V^L and µ = p0, some data distributions cannot be reached by one-step deterministic transport, unlike continuous flows with sample-level flow maps.This contrast motivates the FMLM approach.
  • C.2 Flow maps on the simplex: The simplex-flow construction reparameterizes the mean flow v_s,t through a two-time denoiser δ_s,t that predicts clean data on the simplex.This extends the earlier denoiser-to-velocity parameterization to flow maps.
  • C.2 Flow maps on the simplex: The endpoint denoiser D_t is the posterior mean of clean data given the current noisy point, but it averages posterior multimodality rather than representing the one-step flow map.It remains useful because it matches the geometry of the clean data.
  • C.2 Flow maps on the simplex: For general interpolant coefficients, the velocity field b_t and endpoint denoiser D_t are linked through the conditional expectations defining the interpolant.The derivation uses b_t(x) = E[ẊI_t|I_t = x] and solves for the conditional clean-data mean.
  • C.2 Flow maps on the simplex: For α_t = 1−t and β_t = t, the endpoint denoiser equals a single Euler step of size (1−t), whereas the flow map is the exact ODE solution.This clarifies why denoising and flow-map prediction are distinct objectives.

C.3 The two-time denoiser. · C.4 Characterizing the two-time denoiser · C.5 Learning the two-time denoiser

The two-time denoiser exactly recovers the flow map, remains simplex-valued, matches the standard denoiser on the diagonal, and obeys a semigroup condition. These characterizations yield KL or cross-entropy objectives whose unique critical point is the desired denoiser, for both distillation and self-distillation.

  • C.3 The two-time denoiser.: The two-time denoiser exactly recovers the flow map.This is the first property in Proposition 3.3.
  • C.3 The two-time denoiser.: The two-time denoiser lies on the simplex at every token position.It is shown to be non-negative and normalized because it is a weighted average of single-time denoisers along the flow trajectory.
  • C.3 The two-time denoiser.: On the diagonal, the two-time denoiser recovers the standard denoiser: δs,s(x) = Ds(x).This holds for all x ∈ R^L×|V| and s ∈ [0, 1].
  • C.4 Characterizing the two-time denoiser: The two-time denoiser also satisfies a semigroup condition, with γ = (1−t)(u−s) / ((1−u)(t−s)).For s ≤ u ≤ t, the coefficients form a convex combination.
  • C.4 Characterizing the two-time denoiser: The flow-map characterizations translate into Lagrangian, Eulerian, and semigroup conditions on δs,t.The semigroup condition is explicitly expressed as a convex combination when s ≤ u ≤ t.
  • C.5 Learning the two-time denoiser: For a frozen pre-trained denoiser, Lagrangian, Eulerian, and semigroup distillation losses each uniquely recover the corresponding two-time denoiser.The diagonal term anchors the prediction to the denoiser, while the off-diagonal term enforces the selected characterization.
  • C.5 Learning the two-time denoiser: The semigroup objective has δs,t as its unique critical point under an expectation distribution with full support on 0 ≤ s ≤ u ≤ t ≤ 1.Its teacher is the semigroup teacher, and the objective is KL-based.
  • C.5 Learning the two-time denoiser: The same three objectives support direct self-distillation, with cross-entropy on the one-hot diagonal and unique recovery of δs,t.The diagonal cross-entropy is bounded below by the conditional entropy and is achieved when ˆδt,t = Dt.

C.6 Decoding error rate and entropic time reparameterizations

The section connects the decoding-error time reparameterization to entropic time, which equalizes uncertainty reduction measured by conditional entropy. Under a uniform-token assumption, Fano’s inequality yields an asymptotic relationship with vanishing vocabulary-dependent margin, while non-uniform data weakens the guarantee and online entropic time empirically underperformed.

  • Connection: The decoding-error and entropic reparameterizations are related through Fano’s inequality, which links decoding-error probability with conditional entropy.The result is formulated as an approximate asymptotic inequality with a vanishing margin.
  • Entropic time: Entropic time reparameterization allocates equal uncertainty reduction to each timepoint, measured by clean-data conditional entropy given the noisy state.Dieleman et al. estimate token-level conditional entropy online using the denoiser’s training loss.
  • Proposition C.12: Under uniformly distributed tokens, the proposition derives the relationship between entropic time and decoding-error time as vocabulary size grows.The proof compares the error-based reparameterization with its large-vocabulary limit and bounds the remaining term by O(1/log |V|).
  • Limitations: For realistic non-uniform token frequencies, the proposition’s strong uniformity assumption changes the bound to σ + O(1/log |V|) ≥ 1 − cP_e for some c > 1.The passage specifically identifies Zipf-like distributions as a more realistic alternative.
  • Empirical comparison: Online entropic time based on prior work empirically underperformed the authors’ decoding-error time reparameterization.The theoretical relationship nevertheless indicates that the two time parameterizations are closely connected.

D Two-stage flow map distillation with squared loss

Two-stage squared-loss distillation first corrects Euler flow-map estimates with a residual model, then compresses the resulting two-model map into a single model. The approach is structurally principled and reasonably effective, but does not exploit one-hot geometry and is outperformed by cross-entropy distillation.

  • Motivation: The squared-loss scheme is more cumbersome than cross-entropy distillation and does not use one-hot geometry, which contributes to its weaker empirical performance.The authors nevertheless characterize it because it performs reasonably well and may be useful for future work.
  • First stage: The first stage learns a residual correction to Euler’s flow-map estimate, enforcing the semigroup condition while preserving boundary and tangent conditions by construction.The correction model is initialized from the pretrained FLM after removing its output softmax and zeroing the final layer.
  • Second stage: The second stage distills the two-model flow map into a single-model map, reducing inference memory from the composed teacher’s two models to one.The frozen teacher supplies targets through one forward pass, and the regression objective has a unique global minimizer and strong convexity in the student.
  • Results: On LM1B and OWT, the first-stage squared-loss model achieves performance comparable to cross-entropy distillation, despite using twice the inference memory.The reported comparison is in Table 24; the provided passage does not include its numerical entries.

E Implementation details

FLM is trained from scratch on LM1B and OWT for 1M steps with shared optimization settings, while sampling uses Euler for FLM and γ-sampling for FMLM. The implementation also uses lookup-table spline interpolation for time reparameterization and established distilled or official checkpoints for baseline comparisons.

  • Time reparameterization: Time reparameterization uses a precomputed 1,000-point lookup table, Gauss-Hermite quadrature, and spline interpolation instead of evaluating the probability sum during training.The table approximates the CDF and stores (t, τ) pairs on an equispaced grid over t ∈[0, 1].
  • Training details: FLM training on LM1B and OWT uses 1M steps, batch size 512, 2,500 warmup steps, and a constant learning rate of 3 × 10−4.Adam uses β1 = 0.9 and β2 = 0.999, with softcapping for numerical stability.
  • Sampling details: FLM sampling uses an Euler solver, whereas FMLM uses Kim et al.’s γ-sampling algorithm with optimal γ values on both datasets.These sampling choices are reported for both LM1B and OWT.
  • Many-step baselines: Baseline comparisons use reproduced or official checkpoints, with RDLM unavailable for OWT because no official checkpoint existed and resources were limited.LM1B baselines Duo, MDLM, and CANDI were trained from scratch using identical settings, while RDLM used its official 1M-step checkpoint.
  • Few-step baselines: Few-step baselines include SDTT applied to MDLM and DCD applied to Duo, each using 8 distillation rounds of 10k training steps.Di4C uses intermediate checkpoints selected for best 32-step performance, corresponding to 20k LM1B steps and 50k OWT steps.

F Supplementary evaluation results

Supplementary evaluations find that one-step FMLM generations retain substantial diversity without mode collapse, while Gaussian priors and semigroup-based flow-map distillation are favored in ablations. FMLM also matches discrete diffusion asymptotically in computational complexity and achieves near-perfect Sudoku uniqueness and novelty under 1024-step generation.

  • Diversity and mode collapse: FMLM does not exhibit mode collapse in one-step generation, attaining a Self-BLEU score only slightly worse than real data.Mode collapse would correspond to a Self-BLEU score ≈1.0.
  • Diversity and mode collapse: GPT-4.1 judges FMLM as less diverse than real data, with win rates of 39% and 42%, but clearly not mode-collapsed.A 50% win rate would indicate diversity matching real data; evaluation uses randomly sampled pairs and swaps presentation order to mitigate positional bias.
  • Ablations: The Gaussian prior p0 = N(0, I) outperforms the uniform simplex prior p0 = Dir(1) in the LM1B ablation.The paper attributes the uniform prior’s underperformance to high-dimensional simplex concentration interacting with time reparameterization; for vocabulary size |V|, Dir(1) component variance is ∼1/|V|^2.
  • Ablations: The semigroup-based flow-map objective provides stable training and generation quality, whereas Eulerian training is highly unstable and Lagrangian training is more stable but worse.Eulerian instability is attributed to computing the spatial Jacobian through the full L × |V| state.
  • Computational cost: FLM has O(L|V|) memory and O(L|V|d) forward-pass time, matching the asymptotic complexity of discrete diffusion models.FLM adds only a single input linear projection to lift the |V|-dimensional one-hot vector to d dimensions.
  • Sudoku generation: FMLM achieves near-perfect uniqueness and novelty on 1024-step Sudoku generation, with validity, uniqueness, and novelty coinciding because all generated grids are unique and valid.The evaluation compares FMLM, MDLM + SDTT, and Duo + DCD over 1024 generated Sudoku grids.

G Supplementary qualitative results

Supplementary samples show that FMLM produces coherent, prefix-aligned one-step text and that FLM/FMLM deterministic refinement recovers lexical detail as sampling steps increase. Eight-step FMLM guidance also steers generations toward safety and sports-topic attributes while preserving sample quality.

  • Qualitative samples: FLM samples are shown across 32, 128, 256, and 1024 steps on LM1B and across 256 and 1024 steps on OWT.The supplementary figures provide qualitative comparisons across sampling-step counts.
  • Samples from fixed initial noise: Deterministic FLM and FMLM sampling recovers finer lexical details while preserving general structure as sampling steps increase, unlike the reported discrete baselines.The comparison uses fixed initial random noise and includes MDLM + SDTT and Duo + DCD.
  • Inference-time guidance: Eight-step FMLM+FMRG generations follow safety and sports-topic guidance well while preserving sample quality.The safety reward targets the non-offensive class, and the topic reward targets the sports class.
  • Qualitative sample metrics: 90.94 is FMLM’s Gen.PPL at one step in the reported sample, compared with 1544.76 for MDLM+SDTT and 4726.65 for DUO+DCD.The corresponding entropies are 4.11, 5.39, and 5.97, respectively.
  • Qualitative samples: FMLM generates grammatically correct, prefix-aligned one-step conditional samples on OWT, whereas distilled discrete diffusion baselines fail to produce proper or aligned sentences.Figure 33 compares FMLM with distilled discrete diffusion baselines.
Loading 2602.16813v3…