CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Junzhe Shen; Jieru Zhao; Ziwei He; Zhouhan Lin

Source-linked AI summary

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

Junzhe Shen, Jieru Zhao, Ziwei He, Zhouhan Lin

arXiv:2603.02547v1Mar 3, 2026cs.CLcs.AIcs.LG

TL;DR

Continuous diffusion language models have lagged behind discrete methods because mapping denoised continuous embeddings to discrete tokens is a bottleneck. CoDAR keeps diffusion continuous and uses a context-aware autoregressive decoder for rounding, improving generation over latent diffusion and becoming competitive with discrete DLMs.

Problem
Continuous diffusion language models lag behind discrete approaches because continuous representations must be mapped back to categorical tokens, while existing rounding can be limited by sequence dependence and local evidence.
Method
CoDAR factorizes generation into continuous diffusion over embedding sequences and an autoregressive Transformer decoder that cross-attends to denoised states and produces tokens.
Results
CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs on LM1B and OpenWebText.
Takeaways & Limitations
Decoder temperature provides a simple way to navigate CoDAR’s fluency–diversity trade-off while keeping the model and procedure fixed.
Takeaways & Limitations
The framework assumes a fixed pretrained text embedding model E(·), with only the diffusion and autoregressive decoder parameters optimized.

Abstract

from arXiv · show

We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.

1. Introduction

Continuous diffusion language models face a fundamental continuous–discrete mismatch, with token rounding emerging as a principal bottleneck. CoDAR addresses this by combining continuous embedding diffusion with contextual autoregressive decoding.

1. Introduction: Continuous DLMs lag behind discrete approaches because continuous diffusion must map representations onto inherently categorical linguistic tokens.Prior approaches use rounding, logit-space diffusion, or latent diffusion to bridge this mismatch.
1. Introduction: High-dimensional latent representations can improve reconstruction while complicating diffusion training and generation quality.The paper notes analogous capacity-related pressures for language embedding sequences.
1. Introduction: CoDAR combines continuous diffusion in embedding space with an autoregressive Transformer decoder that contextually rounds embeddings into tokens.The decoder cross-attends to denoised states and performs the discrete realization step.
1. Introduction: Token rounding is identified as a principal bottleneck for continuous embedding DLMs, especially under low-dimensional hidden states.Controlled token-recovery experiments support the limitation of pointwise classifiers.
1. Introduction: CoDAR improves generation quality over latent diffusion and closes the gap with discrete diffusion language models.The contribution is stated as a two-stage framework with continuous generation and contextual token recovery.

2. Related Work

Prior work explores embedding diffusion, diffusion–autoregressive hybrids, and joint continuous–discrete formulations. These approaches seek to combine continuous latent expressivity with better decoding, trainability, or sequence structure.

2. Related Work: Embedding-space methods diffuse continuous word or latent representations, with efforts addressing self-conditioning, optimization pathologies, and decodability.Examples include Diffusion-LM, Self-conditioned Embedding Diffusion, Difformer, and latent diffusion for language generation.
2. Related Work: Hybrid diffusion–autoregressive architectures combine diffusion refinement or parallelism with autoregressive structure and fluency.AR-Diffusion introduces position-dependent denoising, while DGLM generates continuous representations with diffusion.
2. Related Work: Continuous–discrete hybrids jointly model tokens and continuous states to preserve latent expressivity while improving trainability or conditional structure.CCDD, CANDI, and CADD use different couplings between discrete corruption and continuous representations.

3. Theoretical Analysis

The analysis argues that pointwise rounding is suboptimal because token recovery depends on sequence-level context and full-sequence evidence. Controlled experiments support contextual autoregressive decoding as a stronger alternative.

3. Theoretical Analysis: Pointwise decoders incur a locality gap and cannot exploit sequence-level coupling, whereas sequence decoders can condition on the full continuous representation.The formal analysis separates residual token dependence from uncertainty caused by observing only each local vector.
3. Theoretical Analysis: Even larger embedding dimensions reduce local ambiguity without making token recovery pointwise, because off-manifold states may match multiple globally coherent sequences.Global context remains necessary when diffusion outputs are imperfect denoised embeddings.
3. Theoretical Analysis: The controlled study compares a position-wise linear classifier with an autoregressive Transformer that cross-attends to the full hidden-state sequence.Token recovery is measured by whether each predicted token matches the true token.
3. Theoretical Analysis: 0.841 versus 0.154 at d=64, and 0.915 versus 0.297 at d=768: the Transformer recovers tokens far more accurately than the Linear baseline.The comparison uses token recovery accuracy across low- and high-dimensional representations.
3. Theoretical Analysis: These findings motivate decoupling continuous diffusion generation from contextual token rounding in a two-stage architecture.The diffusion model generates continuous embeddings, while the autoregressive decoder maps them back to discrete tokens.

4. Continuous Diffusion with Contextual AutoRegressive Decoder

CoDAR keeps diffusion in a continuous embedding space and uses a contextual autoregressive Transformer decoder to recover discrete tokens. The decoder is trained for robustness to imperfect denoised embeddings, producing a two-stage generation pipeline that can use pretrained text embeddings without an encoder–decoder language model.

Continuous Diffusion with Contextual AutoRegressive Decoder: CoDAR runs diffusion entirely in continuous embedding space, then uses a cross-attending autoregressive Transformer decoder to translate denoised embeddings into tokens.The embedding model is frozen by default, while the denoiser and decoder are optimized.
Continuous Diffusion for Embedding Generation: The diffusion component uses a variance-preserving process with velocity prediction to generate denoised embedding sequences.The process corrupts embeddings with Gaussian noise and uses a cosine schedule; velocity parameterization is motivated by stability during few-step sampling.
Contextualized Rounding with AR Decoder: The decoder learns contextualized token recovery by cross-attending to recovered embeddings and factorizing token likelihood autoregressively.Training uses teacher forcing with a standard cross-entropy objective.
Contextualized Rounding with AR Decoder: Unlike LD4LG, CoDAR does not require an encoder–decoder language model and can use off-the-shelf pretrained text embedding models.The distinction concerns the model used to represent and decode the continuous latent sequence.
Contextualized Rounding with AR Decoder: Noise augmentation perturbs recovered embeddings during decoder training, encouraging robustness to denoiser deviations and alignment with diffusion outputs.Small Gaussian perturbations are added to the conditioning embeddings before decoding.
Contextualized Rounding with AR Decoder: At inference, CoDAR first reverses diffusion from Gaussian noise to obtain embeddings, then autoregressively decodes tokens while cross-attending to them.This separates continuous generation from discrete realization into two sequential stages.

5. Experiment

Experiments evaluate CODAR on unconditional generation across datasets, samplers, step budgets, hidden dimensions, and decoder architectures. CODAR offers a controllable fluency–diversity trade-off, benefits from DPM-Solver for fast sampling, and shows that contextual decoding and lower-dimensional states are important for quality.

5.2. Main Results: At T=1.00, CODAR reaches diversity 0.4842, matching or slightly exceeding MDLM (0.4784) and SEDD (0.4742), while offering better fluency at lower temperatures.On OpenWebText, CODAR's Gen. PPL is 47.71 at T=0.00 and 50.68 at T=0.25, with non-trivial diversity of 0.1660–0.1937.
5.3.1. Sampler and Sampling Steps: DPM-Solver improves fluency across sampling budgets while maintaining high diversity, reducing Gen. PPL from 185.91 to 154.83 at 100 steps.At 25 steps, it slightly improves Gen. PPL to 212.32 from 214.86 while preserving strong diversity.
5.3.1. Sampler and Sampling Steps: At 25 steps, CODAR achieves the best fluency among compared methods while retaining strong diversity; at 50–250 steps, its diversity remains around 0.49.At 250 steps, CODAR reaches 0.488 diversity versus 0.478 for MDLM and 0.474 for SEDD.
5.3.1. Sampler and Sampling Steps: Increasing hidden dimension worsens quality: Gen. PPL rises from 164.90 at d=64 to 294.42 at d=256 and 523.07 at d=768.The experiment attributes this degradation to larger hidden states hindering the diffusion process despite increasing decoder representational capacity.
5.3.1. Sampler and Sampling Steps: The Transformer decoder yields diversity 0.4842 versus 0.1238 for a linear decoder, despite the linear decoder's slightly lower Gen. PPL of 153.44 versus 164.90.The linear decoder's low diversity indicates severe repetition and mode collapse.

6. Conclusion

The conclusion attributes the gap between continuous and discrete diffusion language models primarily to decoding and rounding rather than continuous diffusion itself. CODAR addresses this with continuous embedding diffusion and contextual autoregressive decoding, achieving stronger generation quality and controllable fluency–diversity trade-offs.

6. Conclusion: CODAR treats rounding as a contextual sequence problem, combining continuous embedding diffusion with an autoregressive Transformer decoder.The conclusion reports theoretical analysis and controlled token-recovery experiments supporting this design.
6. Conclusion: Experiments on LM1B and OpenWebText show that CODAR improves over latent diffusion and becomes competitive with strong discrete DLMs.The decoder temperature provides a simple way to trade fluency against diversity.

A. More Experimental Details

The additional experimental details specify separate optimization settings for the contextual decoder and DiT diffusion model.

A. More Experimental Details: The decoder uses Adam with learning rate 1.0 × 10^-3, weight decay 1.0 × 10^-1, cosine annealing, and 5% warmup.It is trained for 1 epoch with batch size 512 and gradient clipping at norm 1.0.
A. More Experimental Details: The DiT uses Adam with batch size 512, learning rate 4.0 × 1.0^-4, a constant schedule, and weight decay 0.02.

B. Proof

The proof characterizes Bayes-optimal sequence and pointwise decoders, then shows their optimality gap is lower-bounded by conditional total correlation and is nonnegative.

Step 1: Bayes-optimal decoder over Dseq: The proof begins by establishing the Bayes-optimal decoder over full token sequences through conditional cross-entropy decomposition and KL nonnegativity.The candidate decoder is optimized separately for each fixed conditioning value.
Step 2: Bayes-optimal decoder over Dpw: It then restricts the decoder to pointwise-factorized predictions, for which the sequence log-likelihood separates across positions.Each position can be analyzed using the joint law of its local input and token.
Step 3: The exact optimality gap: The difference between the two optimal risks yields an exact optimality gap between full-sequence and pointwise decoding.The equality follows by subtracting the two minima obtained in the preceding steps.
Step 4: Lower bound by conditional total correlation and nonnegativity: The gap is at least the conditional total correlation, which is nonnegative because it is tied to a KL divergence and conditioning reduces entropy.The proof concludes by applying the definition of conditional total correlation and the entropy comparison between full and local conditioning.