PaperlayerView exact revision on arXiv

Source-linked AI summary

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu

arXiv:2604.11748v3cs.CLcs.LG

TL;DR

Embedding-space diffusion language models lacked a reliable theoretical evaluation framework and clear training-design choices. LangFlow addresses these gaps with Flow Matching, an ODE-based NLL bound, information-uniform scheduling, and self-conditioning, achieving competitive results with discrete diffusion and beating autoregressive baselines on 4 of 7 zero-shot benchmarks.

  • Problem

    Embedding-space diffusion language models lacked reliable ODE-based perplexity evaluation and clear, theoretically grounded training-design choices.

  • Method

    LangFlow grounds embedding-space diffusion in Flow Matching via Bregman divergence, adding an ODE-based NLL bound, Gumbel noise scheduling, and self-conditioning.

  • Results

    LangFlow matches state-of-the-art masked diffusion, surpasses uniform-state discrete diffusion, and beats autoregressive baselines on 4 of 7 zero-shot benchmarks, with PPL 30.0 on LM1B.

  • Takeaways & Limitations

    These results provide consolidated evidence that continuous diffusion can be competitive with discrete diffusion for language modeling.

  • Takeaways & Limitations

    Sample entropy remains lower than that of certain discrete baselines, with possible effects at larger scales left for future investigation.

Abstract

from arXiv · show

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

1 Introduction

LangFlow addresses the unresolved challenges of embedding-space diffusion language models by grounding them in Bregman Divergence Flow Matching and introducing principled evaluation and improved design choices. It achieves competitive language-modeling results, including PPL of 30.0 on LM1B and 24.6 on OpenWebText, while exceeding autoregressive baselines on 4 of 7 zero-shot benchmarks.

  • Motivation: Continuous diffusion offers expressive latent trajectories, stable ODE/SDE sampling, and techniques such as self-conditioning, trajectory editing, and flow-based acceleration.These capabilities have driven strong results across images, videos, molecular structures, and robot behaviors.
  • Motivation: Embedding-space diffusion avoids categorical sparsity while enabling editable generation paths through progressively denoised token embeddings.This makes it a promising but underexplored direction for language modeling.
  • Challenges: Prior embedding-space DLMs lack clear theoretical grounding and reliable ODE-based perplexity estimation, with some training objectives being heuristic or cumbersome.One prior approach requires dynamically sliced batches to optimize different objectives.
  • Method: LangFlow connects embedding-space DLMs to Flow Matching through Bregman divergence, yielding a theoretically grounded cross-entropy loss and a novel ODE-based NLL upper bound.The bound addresses evaluation challenges and is presented as more accurate than prior SDE-based ELBOs.
  • Results: 30.0 PPL on LM1B and 24.6 PPL on OpenWebText establish LangFlow as a continuous DLM that exceeds uniform-state discrete diffusion and matches state-of-the-art masked diffusion.LangFlow also beats autoregressive baselines on 4 out of 7 zero-shot benchmarks and masked diffusion on 3 out of 7.

2 Preliminaries

Flow Matching learns a velocity field that transports a simple prior to the data distribution by solving an ODE. In practice, the marginal field is approximated with conditional Flow Matching, using affine Gaussian paths and equivalent denoiser parameterization.

  • Flow Matching: Flow Matching learns a velocity field u_t(z_t) that transports a simple prior p_prior to p_data by solving d z_t = u_t(z_t) d t.Starting from z_0 ∼ p_prior, the ODE solution yields z_1 ∼ p_data.
  • Flow Matching: The marginal velocity field is obtained by marginalizing conditional velocity fields that transform the prior into specific data points.Under mild regularity conditions, this marginal field generates p_data.
  • Flow Matching: Typical Flow Matching models use affine Gaussian probability paths with differentiable schedules α_t and σ_t satisfying α_0 = 0, σ_0 = 1, α_1 = 1, and σ_1 = 0.The schedules specify the transformation and have time derivatives ˙α_t and ˙σ_t.
  • Flow Matching: Because the true marginal velocity field is intractable, a neural network v_θ(z_t, t) is trained using conditional Flow Matching.The optimization omits a constant term independent of θ.
  • Flow Matching: Since the conditional velocity is linear in z, the velocity network can equivalently be parameterized by a denoiser ẑ_θ(z_t, t) approximating E[z | z_t].With the consistently used standard Gaussian prior, Flow Matching and diffusion models are treated interchangeably.

3 LangFlow: Continuous Language Modeling via Flow Matching

LangFlow presents a principled continuous diffusion language model that trains in token space with cross-entropy while deriving a continuous denoiser for flow-based sampling. Its γ-path parameterization and ODE-based likelihood evaluation reduce schedule dependence and provide deterministic generation.

  • Evaluation and sampling: LangFlow derives a novel ODE-based upper bound for estimating perplexity, replacing prior stochastic likelihood bounds with evaluation integrated along the continuous flow path.The framework uses deterministic ODE sampling, preserving a bijective prior-to-data mapping for potential few-step distillation.
  • Training objective: LangFlow connects cross-entropy training for categorical diffusion with Flow Matching through Bregman-divergence minimization.Cross-entropy is established as a principled special case for categorical data.
  • γ-path and pipeline: The model embeds token sequences into a continuous space, perturbs embeddings along a γ-path, and predicts token probabilities from noisy embeddings.The γ variable is the logarithmic noise-to-signal ratio, making noise level rather than an arbitrary time index the conditioning variable.
  • γ-path and pipeline: The γ-path reparameterizes valid diffusion trajectories by logarithmic noise-to-signal ratio, reducing dependence on a specific time schedule.The implementation uses γ as the time-conditioning variable instead of t.
  • Continuous denoiser: Predicted token probabilities deterministically define the continuous denoiser through their embedding expectation, coupling discrete likelihood training to continuous ODE flow estimation.This denoiser supports ODE-based sampling from the associated probability predictions.

4 Improved Design Choices of Continuous Diffusion Language Models

Section 4 identifies noise scheduling and self-conditioning as key design choices for continuous diffusion language models. Information-uniform Gumbel scheduling and self-conditioning together enable LangFlow to match discrete diffusion in perplexity and sample quality.

  • LangFlow’s two key design choices are noise scheduling and self-conditioning, which together match discrete diffusion in perplexity and sample quality.
  • 4.1 Noise Scheduling: Uniform t-based scheduling wastes more than half of training and sampling steps because loss is nearly zero for t ∈[0.2, 1.0].
  • 4.1 Noise Scheduling: The information-uniform principle allocates noise density according to information gain per noise-level change, concentrating training and sampling where H′γ is largest.The principle follows from viewing diffusion sampling as progressively reducing posterior entropy.
  • 4.1 Noise Scheduling: 154.2: Gumbel-quantile scheduling reduces LangFlow’s generative perplexity from ∼1000 to 154.2.The scheduler uses learnable Gumbel parameters and samples intermediate noise levels at Gumbel quantiles.
  • 4.2 Self-Conditioning: Self-conditioning improves both PPL and Gen. PPL for LangFlow, unlike MDLM, where it improves Gen. PPL but degrades PPL.This asymmetry makes disabling self-conditioning during continuous-diffusion PPL evaluation unfair and makes it crucial for LangFlow’s performance.

5 Experiments

Experiments evaluate LangFlow against autoregressive, discrete diffusion, and continuous diffusion baselines on LM1B and OpenWebText using perplexity, generative perplexity, and zero-shot transfer. LangFlow matches state-of-the-art discrete DLMs on language modeling and shows strong downstream transfer, ranking first on several OWT-based tasks.

  • Experimental Setup: LangFlow is evaluated on LM1B and OpenWebText, following established language-modeling benchmarks and sequence-packing practice.
  • Evaluation: Evaluation reports validation perplexity and GPT-2-Large generative perplexity from 1024 generated samples, using 128 sampling steps on LM1B and 1024 on OpenWebText.
  • Experimental Setup: The comparison includes an autoregressive Transformer, six discrete DLMs, and continuous baselines Diffusion-LM and Plaid.The discrete baselines are D3PM, DiffusionBert, SEDD, MDLM, UDLM, and Duo.
  • Language Modeling: LangFlow achieves the best PPL on LM1B, the third-best PPL on OpenWebText, and ranks second in Gen. PPL on LM1B and first on OpenWebText.
  • Zero-Shot Transfer: In zero-shot transfer from OpenWebText, LangFlow ranks first on PTB, Wikitext, and Lambada while remaining competitive across the other evaluated domains.The downstream corpora also include LM1B, AG News, PubMed, and Arxiv.

6 Additional Related Work

Prior work spans discrete diffusion, continuous diffusion on token-probability simplexes and embeddings, tractable NLL bounds, heuristic noise scheduling, and variational flow matching. LangFlow is positioned against these approaches through its Bregman-divergence-based derivation and theoretically grounded alternatives.

  • Discrete Diffusion Language Models: Discrete diffusion models categorical states and support scalable language modeling with parallel decoding.
  • Continuous Diffusion Language Models: Simplex-based continuous diffusion struggles to learn score functions at scale because sparse signals exacerbate the curse of dimensionality.
  • Continuous Diffusion Language Models: Existing embedding-space DLM objectives are often heuristic, making rigorous negative log-likelihood evaluation difficult.
  • Continuous Diffusion Language Models: Plaid offers a tractable SDE-based NLL upper bound, but optimization requires dynamically sliced batches to balance loss terms.
  • Noise Scheduling: Importance sampling schedules training by loss distributions over noise levels, whereas LangFlow allocates budget according to loss-derivative distributions under a theoretically grounded principle.
  • Variational Flow Matching: Variational Flow Matching uses auxiliary posteriors, while Exponential-Family VFM extends variational objectives through Bregman-divergence-based moment matching.

7 Conclusion

LangFlow establishes a principled foundation for continuous diffusion language modeling and achieves strong benchmark performance, including PPL of 30.0 on LM1B and 24.6 on OpenWebText. Its remaining limitation is lower sample entropy than certain discrete baselines, without noticeable sample-quality degradation in the reported evaluations.

  • Contributions: LangFlow grounds embedding-space diffusion language modeling in Flow Matching via Bregman divergence, with a cross-entropy objective, ODE-based NLL bound, information-uniform scheduling, and self-conditioning.These components form a coherent foundation for continuous diffusion in language modeling.
  • Empirical results: 30.0 PPL on LM1B and 24.6 PPL on OpenWebText demonstrate LangFlow’s strong large-scale benchmark performance.LangFlow surpasses uniform-state discrete diffusion and matches state-of-the-art masked diffusion at the same model and data scale.
  • Empirical results: LangFlow exceeds autoregressive baselines on multiple zero-shot transfer benchmarks, providing consolidated evidence for continuous diffusion in language modeling.The paper describes this as the first consolidated evidence that continuous diffusion can fully realize its advantages in language.
  • Limitations & Future Work: LangFlow’s sample entropy remains lower than that of certain discrete baselines, although evaluations found no noticeable sample-quality degradation attributable to this difference.Subtle effects of reduced entropy may emerge at larger scales.

Appendices … A.3 Variational View of LangFlow

The appendices specify LangFlow’s training and Euler-sampling procedures, explain the solver assumption, and show that its objective has a variational interpretation as the negative log-likelihood of a factorized approximation to the endpoint posterior.

  • A.1 Algorithms: LangFlow’s complete training and sampling procedures are summarized in Algorithms 1 and 2.The training procedure includes optimization of the combined LCE + LScheduler objective until convergence.
  • A.1 Algorithms: Self-conditioning is applied stochastically during training by predicting ˆx, converting it to embeddings ˆz, and stopping its gradient.When self-conditioning is not selected, ˆz is set to zero.
  • A.1 Algorithms: The scheduler regularizes the clipped cross-entropy loss toward Hγ and optimizes LCE + LScheduler.Hγ uses a Gumbel-shaped expression, Hγ = H+∞· exp(−exp(−(γ − Pµ)/Pβ)).
  • A.2 Numerical Solver: The Euler solver assumes the denoised embedding ˆzθ(zγ, γ) remains constant within each small integration interval.This assumption aligns the numerical solver with LangFlow’s sampling design.
  • A.3 Variational View of LangFlow: LangFlow also admits a variational interpretation in which its training objective is exactly the negative log-likelihood of a factorized approximation to p(x | zγ).The noisy state is zγ = αγz + σγϵ with ϵ ∼ N(0, I), and the variational factors are parameterized by model predictions.

B Proofs … C.1 Model architecture

The paper proves an ODE-based variational lower bound for continuous flow-based language models by treating the least-noisy state as a latent variable. It then specifies a roughly 130M-parameter DiT architecture with self-conditioning, normalized embeddings, and tokenwise bias corrections.

  • B.1 Theorem 3.1: The proof begins with a VAE-style ELBO that treats the least noisy continuous state z_a as a latent variable for discrete data x.The prior p_a(z_a) is defined for this latent state, while θ(z_a, a) serves as the decoder.
  • B.1 Theorem 3.1: The variational term E_p(z_a|x)[−log p_a(z_a | x)] is evaluated using the entropy of a Gaussian distribution.
  • B.1 Theorem 3.1: The proof computes log p_a(z_a) with the Instantaneous Change of Variables formula and integrates the resulting expression.
  • B.1 Theorem 3.1: The resulting bound is log p(x) ≥ E_p(z_a|x)[log p_a(z_a) + log p̂(x | z_a)] + L_D.
  • B.1 Theorem 3.1: The proof relates velocity and denoising through Equation (6), enabling computation of the divergence term before concluding the theorem.
  • C.1 Model architecture: The AR, SEDD, MDLM, UDLM, and Duo baselines use the same 130M modified DiT architecture, while each retrained baseline retains its own default setup.Plaid is trained with its own codebase, and UDLM uses its codebase.
  • C.1 Model architecture: Self-conditioning updates the main input as z_γ ← z_γ + W_in z_γ + W_SC z_SC before the DiT blocks, with W_in and W_SC zero-initialized.The architecture also normalizes embeddings onto a unit sphere and scales them by √768 to align data variance with noise variance.
  • C.1 Model architecture: A Plaid-inspired tokenwise bias r log p(z^(i)_γ | x^(i)) is added to predicted logits, with r ramped from 0 to 1 over the first 5000 iterations.These modifications leave the model at around 130M parameters through training.

C.2 Training Details

LangFlow training uses dataset-specific GPU setups with bfloat16 precision, while sharing the same AdamW optimization and learning-rate schedule across LM1B and OWT.

  • C.2 Training Details: LM1B training uses 4 NVIDIA RTX 6000 GPUs, while OWT training uses 32 NVIDIA A100 GPUs, both with bfloat16 precision.Both datasets use AdamW with a learning rate of 3 × 10−4, EMA decay of 0.9999, and a constant schedule after 2,500-step linear warmup.
  • C.2 Training Details: Both datasets use AdamW with a 3 × 10−4 learning rate, 0.9999 EMA decay, and a constant schedule after 2,500-step linear warmup.These optimization settings are shared across LM1B and OWT.

C.3 Plaid Baseline … D.1 Self-Conditioning Dynamics

The Plaid baseline differs architecturally from LangFlow and other baselines, while its MSE-based training objective produces token-embedding collapse. Additional evaluation uses specified numerical procedures, and self-conditioning shifts posterior uncertainty from semantic alternatives toward frequent syntactic tokens as noise increases.

  • C.3 Plaid Baseline: Plaid uses its own Transformer architecture, adding time-condition embeddings to token embeddings with dimension 16 rather than the 768 dimensions used by LangFlow and other baselines.This corresponds to 108M parameters for Plaid versus 130M for the other models.
  • C.3 Plaid Baseline: MSE denoising regression empirically collapses the token embedding layer, clustering embeddings for different vocabulary tokens.The objective pushes incorrect-token embeddings toward the ground-truth embedding and pulls the ground-truth embedding toward a weighted mean of predicted alternatives.
  • C.3 Plaid Baseline: 0.058 is Plaid’s average nearest-neighbor distance among token embeddings, substantially smaller than those of the other compared models.Its nearest-neighbor distribution is also concentrated in the lower half of the range, consistent with mode collapse.
  • C.4 Additional Evaluation Details: Perplexity evaluation uses a 128-step Heun-2 solver to compute the log-likelihood integral and initial state, with Hutchinson’s trace estimator for divergence.These are the stated numerical procedures for evaluating the model’s PPL.
  • C.4 Additional Evaluation Details: Entropy evaluation uses token frequencies from the same 1024 sequences as Gen. PPL, assigning each unique token probability 1/128 for LM1B and 1/1024 for OWT.The assignment applies to each unique token appearing in a sequence.
  • D.1 Self-Conditioning Dynamics: Self-conditioning analysis compares token-posterior evolution across noise levels γ between a first pass without SC and a second pass with SC on LM1B.The representative example uses the ground-truth token “run”.
  • D.1 Self-Conditioning Dynamics: As γ increases, posterior mass progresses from “run” to related alternatives such as “go”, then to frequent syntactic tokens including “is”, “and”, and “the”.The paper interprets this as a transition from semantic uncertainty to frequency-dominated uncertainty.
  • D Additional Experiments: Self-conditioning on the LM1B checkpoint prevents iterative refinement from drifting toward high-frequency but semantically weak tokens.This conclusion follows from the observed posterior progression across noise levels.

D.2 Quantitative Sample Quality · D.3 Qualitative Samples · D.3.1 LM1B

LangFlow’s sample-quality analysis evaluates ODE sampling budgets and examines why its lower entropy does not necessarily indicate poorer semantic quality. Qualitative LM1B samples use fixed-length sequences and motivate caution when interpreting entropy alone.

  • D.2 Quantitative Sample Quality: LangFlow’s NFE ablations vary ODE solver steps on the same trained checkpoint while holding other sampling settings fixed.No distillation or specialized few-step training is used.
  • D.2 Quantitative Sample Quality: On LM1B, the NFE comparison reports LangFlow alone because comparable sweeps from prior continuous and discrete diffusion baselines are unavailable.On OWT, LangFlow is compared with Duo, MDLM, and SEDD under different NFEs.
  • D.2 Quantitative Sample Quality: Table 5 compares LangFlow’s Gen. PPL and entropy across different numbers of function evaluations on LM1B.The table reports both metrics under varying NFEs.
  • D.2 Quantitative Sample Quality: On OWT, LangFlow has lower sample entropy than baselines at the same sampling-step count, which can suggest repetition that harms semantic quality.The analysis then examines whether this interpretation is warranted.
  • D.2 Quantitative Sample Quality: The entropy reduction primarily reflects frequent content words rather than undesirable local repetition.This distinguishes global frequency effects from repeated adjacent or nearby phrasing.
  • D.2 Quantitative Sample Quality: A selected LangFlow sample contains a single content word repeated up to 11 times, and maximum-frequency rankings align with sample-entropy rankings.The analysis counts each sample’s most frequent content word using samples from Section D.3.
  • D.2 Quantitative Sample Quality: Repeated words are often distributed across distant sentences, suggesting a global frequency bias rather than local degeneration and limiting their impact on semantic quality.The phenomenon appears in most LangFlow samples and also in other continuous DLMs.
  • D.3.1 LM1B: Qualitative samples compare language-modeling paradigms using a fixed sequence length of 128 tokens, while entropy should not be treated as a standalone quality measure.Entropy conflates distributional calibration with sequence-level coherence.

LangFlow · D.3.2 OWT · LangFlow

LangFlow’s OWT qualitative evaluation reports generative perplexity and entropy across multiple samples, alongside fixed-length 1024-token generations from different language-modeling paradigms. The reported samples show substantial variation in generative perplexity while entropy remains near 4.2–4.4.

  • LangFlow: 63.4070 Generative Perplexity and 4.3104 Sample Entropy are reported for LangFlow’s first sample.The corresponding sample contains multiple [CLS]-delimited passages of news-like text.
  • LangFlow: 110.6519 Generative Perplexity and 4.3081 Sample Entropy are reported for LangFlow’s second sample.Its generated text spans scientific, sports, and pharmaceutical topics.
  • LangFlow: 67.0254 Generative Perplexity and 4.3124 Sample Entropy are reported for LangFlow’s third sample.The sample includes news-like passages involving crime, baseball, and business reporting.
  • LangFlow: 71.8890, 69.8902, and 91.7882 are the reported Generative Perplexity values for three additional LangFlow samples.Their corresponding Sample Entropy values are 4.3170, 4.3869, and 4.3387, respectively.
  • LangFlow: 133.2676, 60.0276, and 119.1866 are the reported Generative Perplexity values for another three LangFlow samples.The corresponding Sample Entropy values are 4.3306, 4.2995, and 4.4145.
  • LangFlow: 159.5354, 156.5223, and 109.2186 are the reported Generative Perplexity values for three further LangFlow samples.Their corresponding Sample Entropy values are 4.4154, 4.4382, and 4.3417.
  • LangFlow: 181.8081, 89.3953, and 85.1183 are the reported Generative Perplexity values for three additional LangFlow samples.The corresponding Sample Entropy values are 4.3870, 4.2908, and 4.2261.
  • D.3.2 OWT: All qualitative generations in the OWT comparison use a fixed sequence length of 1024 tokens.The section presents samples from different language-modeling paradigms.
Loading 2604.11748v3…