Continuous Latent Diffusion Language Model

Hongcan Guo; Qinyu Zhao; Yian Zhao; Shen Nie; Rui Zhu; Qiushan Guo; Feng Wang; Tao Yang; Hengshuang Zhao; Guoqiang Wei; Yan Zeng

Source-linked AI summary

Continuous Latent Diffusion Language Model

Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang, Hengshuang Zhao, Guoqiang Wei, Yan Zeng

arXiv:2605.06548v1May 7, 2026cs.CLcs.AIcs.CV

TL;DR

Fixed left-to-right token ordering is not necessary for high-quality text generation, but existing alternatives have struggled to combine efficiency, scalable representation learning, and global semantic modeling. Cola DLM addresses this with hierarchical latent diffusion, and experiments show strong generation quality and encouraging scaling behavior under matched comparisons.

Problem
Text generation needs alternatives to fixed token order that support more general state-evolution and denoising paths.
Method
Cola DLM maps text into continuous latents, models their global semantic prior with diffusion, and conditionally decodes local text.
Results
Across 8 benchmarks and matched baselines, Cola DLM shows strong text-generation quality and encouraging scaling behavior.
Takeaways & Limitations
Hierarchical continuous latent prior modeling provides a principled alternative to strictly token-level language modeling.
Takeaways & Limitations
The experiments remain relatively controlled in scale and evaluation, leaving performance at larger models, longer training, and greater compute budgets unexamined.

Abstract

from arXiv · show

Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling. We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition. Cola DLM first learns a stable text-to-latent mapping with a Text VAE, then models a global semantic prior in continuous latent space with a block-causal DiT, and finally generates text through conditional decoding. From a unified Markov-path perspective, its diffusion process performs latent prior transport rather than token-level observation recovery, thereby separating global semantic organization from local textual realization. This design yields a more flexible non-autoregressive inductive bias, supports semantic compression and prior fitting in continuous space, and naturally extends to other continuous modalities. Through experiments spanning 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs, we identify an effective overall configuration of Cola DLM and verify its strong scaling behavior for text generation. Taken together, the results establish hierarchical continuous latent prior modeling as a principled alternative to strictly token-level language modeling, where generation quality and scaling behavior may better reflect model capability than likelihood, while also suggesting a concrete path toward unified modeling across discrete text and continuous modalities.

1 Introduction

Autoregressive language models have driven major advances but impose sequential inference and a fixed token ordering [6] [22] [39] [45] [60] [102]. Cola DLM addresses this gap with hierarchical continuous-latent modeling that separates global semantic modeling from local textual realization and is evaluated across broad experiments and scaling regimes.

Motivation: Existing autoregressive, discrete-diffusion, and continuous-diffusion methods struggle to jointly achieve generation efficiency, scalable representation, and global semantic modeling.Autoregressive models provide a clear token-level training objective but incur sequential inference costs and impose a strong fixed-order inductive bias.
Proposed framework: Cola DLM maps text to continuous latents with a Text VAE, models their prior with a block-causal DiT, and generates text through conditional decoding.Its diffusion process is positioned as latent-space prior modeling rather than token-level observation recovery.
Proposed framework: Cola DLM decomposes text generation into global semantic modeling and local textual realization within a unified probabilistic framework.It uses diffusion-based prior modeling in continuous latent space to connect these stages.
Theoretical perspective: A unified Markov-path analysis characterizes Cola DLM’s advantages in global semantic modeling, non-autoregressive inductive bias, and theoretical interpretability.The introduction states that subsequent experiments validate these advantages.
Empirical scope: Experiments span 4 research questions, 8 benchmarks, matched ∼2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs.They identify an effective overall configuration and evaluate the model’s scaling behavior for text generation.
Additional analyses: Additional analyses examine likelihood–generation-quality mismatch, first-block conditioning, latent compression, and a possible bridge from discrete text to continuous modalities such as vision.The introduction presents this bridge as preliminary evidence toward a broader unified generative paradigm.

2 Related Work

Prior language-modeling alternatives trade off generation flexibility, sampling efficiency, and scalable semantic representation. Autoregressive models impose sequential left-to-right generation, while discrete and continuous diffusion approaches address non-autoregressive generation with distinct limitations.

Autoregressive Language Models: Autoregressive language models [56] [92] [101] use token-level maximum likelihood but constrain generation to a fixed left-to-right order and inherently sequential inference.These limitations make them less suitable for infilling, local editing, and global reorganization.
Discrete Diffusion Language Models: Discrete diffusion language models use discrete transition kernels [2] [10] [88] or masking and absorbing states [69, 70, 80, 81, 84, 10…] to enable non-autoregressive generation through denoising.Transition-kernel methods typically sample slowly and do not easily exploit the smooth semantic structure of continuous spaces.
Continuous Diffusion Language Models: Continuous diffusion language models include vocabulary-aligned methods [31] [43] [59] [79] that diffuse directly over one-hot vectors, logit simplexes, or probability simplexes.Their representation dimension scales with vocabulary size, limiting scalability.

3 Continuous Latent Diffusion Language Model

Cola DLM is a hierarchical latent-variable language model that compresses global semantics into a continuous latent prior and delegates local token realization to a conditional decoder. Its two-stage training and prior-transport formulation separate representation learning, global semantic organization, and textual realization without imposing a left-to-right text-generation path.

Hierarchical latent-variable model: Cola DLM separates text modeling into a global continuous latent prior pψ(z0) and conditional discrete-text realization pθ(x | z0), with qϕ(z0 | x) used only for inference.The latent prior generates global semantics, while the decoder realizes text; the encoder is not part of the generative model.
Overall workflow: The workflow uses Text VAE pretraining to establish a stable latent–text correspondence, followed by joint block-causal DiT training that learns the final latent prior.The VAE remains trainable in Stage 2, while Stage 1’s base prior stabilizes the interface rather than serving as the final generative prior.
Unified theoretical framework: Its diffusion path transports the latent prior rather than recovering text observations, so continuity models latent-distribution geometry instead of imposing a left-to-right inductive bias.The distinction from observation-path methods lies in both the path’s state space and its semantic role.
Modeling rationale: The latent decomposition assigns global semantic compression to z0 and local textual realization to the decoder, making latent organization—not denoising itself—the model’s central advantage.The VAE’s masking loss prevents semantic collapse, and its strictly causal encoder and decoder support streaming generation without sequence-length compression.

4 Experiments

Experiments show that Cola DLM benefits from larger, semantically structured latent spaces that evolve jointly with DiT from stable initialization. Performance also depends on calibrated timeshifts and semantic guidance during latent evolution, with empirical trends matching theoretical predictions.

Timeshift Calibration: The optimal timeshift shifts approximately monotonically from 1.0 at d = 16 to 1.7 at d = 64 and 2.3 at d = 128.The same larger-loc trend appears across LAMBADA, MMLU, SIQA, and Task Avg, indicating that dimensionality changes the denoising scale for semantic recovery.
Timeshift Calibration: Empirical timeshift peaks closely follow theoretical predictions across all three latent dimensions, supporting a non-arbitrary relationship between latent dimension and optimal calibration.These results provide empirical support for shared, semantically relevant global structures in Cola DLM’s latent space and for the corresponding condition in Eq. (3.35).
Latent-Space Training: Joint DiT x1 achieves the best final results and strongest scaling, surpassing fixed-VAE training as compute increases while scratch training remains consistently worse.A fixed latent space helps early stability but saturates, whereas effective adaptation requires meaningful pretrained initialization and sufficiently strong, continuous updates.
Latent-Space Geometry: Scratch latents form collapsed, less organized geometries, whereas jointly trained latents produce richer trajectories and more semantically usable structure.Increasing d from 16 to 128 partially alleviates collapse, but cannot replace proper latent-space formation through stable initialization and joint adaptation.
Latent Dimensionality: Average score rises from 8.7 at d = 16 to 11.3 at d = 64 and 11.8 at d = 128, although larger dimensions do not fully resolve latent collapse.The gains are strongest on MMLU and SIQA, while the geometry remains less structured than with jointly trained, stably initialized latents.
Semantic Importance of the Latent Space: Adding BERT loss produces the best final performance across Task Average, LAMBADA, MMLU, and SIQA when the VAE learning-rate ratio is 1.The benefit is limited at ratio 0.01, showing that strong latent evolution requires semantic guidance rather than trainability alone.

5 Discussion

The discussion shows that Cola DLM’s generation quality can diverge from likelihood-oriented PPL, while conditioning, latent compression, and VAE robustness reveal concrete design trade-offs. It also presents preliminary qualitative evidence that the hierarchical latent-prior formulation extends to unified text–image modeling.

Likelihood and generation: Generation quality reflects semantic latent smoothness, whereas likelihood-oriented PPL additionally depends on local probability-density calibration shaped by the VAE logSNR.Decoder probes and posterior hits remain high while prior-hit rates vary, indicating prior misalignment—not decoder failure—as the main issue around the gold latent region.
Conditioning and padding: Clean condition repaint achieves the best performance across tasks, showing that persistent conditioning preserves the prompt region better than partial noisy correction or positional layout alone.Partial repaint weakens performance, smaller guided portions generally degrade it, and additional repaint repetitions provide no stable gains.
Latent compression: Table 6 indicates that latent compression is promising but currently limited by non-divisible sequence boundaries, while aligned even-length prompts can support stronger abstraction and faster generation.With block size 16, patch size 1 covers 16 decoded text tokens per block, whereas patch size 2 covers 32; odd-length prompts can shift compressed prompt latents and propagate errors.
VAE robustness: Figure 13 shows nearly perfect VAE reconstruction at t = 0, very high accuracy in the low-noise regime, and about 0.92 accuracy at t = 250 before heavier-noise degradation.The gradual degradation supports using the VAE latent space as a stable semantic interface rather than a fragile compressed code.
Multimodal extension: Cola DLM naturally extends to unified text–image modeling by organizing cross-modal semantics in a shared continuous latent space and using modality-specific decoders for realization.The prototype is early-stage and qualitative, based on moderate pretraining on in-house 256 and 640 resolution data without extensive curation or supervised fine-tuning.

6 Limitations & Future Prospects

Cola DLM shows initial evidence of feasibility, competitiveness, and promising scaling potential for text generation in continuous latent space, but the work remains an early foundation rather than a finished endpoint. Current findings come from relatively controlled-scale experiments mainly intended to clarify the framework’s key properties.

Cola DLM has initial evidence supporting feasibility, competitiveness, and promising scaling potential for text generation in continuous latent space.
The paper presents Cola DLM as a starting point for further exploration rather than a finished endpoint.
Current experiments are conducted at a relatively controlled scale and primarily clarify the framework’s key properties.

7 Conclusion

Cola DLM presents hierarchical continuous latent diffusion as a principled alternative to strictly token-level language modeling by separating global semantic prior modeling from local textual realization.

Cola DLM decomposes text generation into global semantic prior modeling in latent space and local textual realization through conditional decoding.
The paper frames hierarchical continuous latent diffusion as a principled alternative to strictly token-level language modeling.
Theoretical analysis and experiments consistently suggest that hierarchical information decomposition can benefit text generation.

8 Afterword: Research Objectives and Significance … B.3 Conditional Probability Estimation and Block-Level Conditional Scoring

Cola DLM is presented as a unified redesign of representation, objective, and learning environment: hierarchical continuous latents separate semantic organization from textual realization, while strict probabilistic analysis distinguishes likelihood objectives from practical flow-based solvers and estimators. The appendix further specifies sampling and probability estimation, including block-level conditional scoring and its limitations.

8.1 Rethinking Text Modeling Paradigms: From State Space in the System to Hierarchical Text Generation: Cola DLM replaces direct token-surface modeling with hierarchical latent representation, organizing global semantics in continuous latent space before conditional local text realization.The latent formulation treats Cola DLM as a hierarchical latent-variable language model rather than a reverse process operating directly in token space.
8.2 Understanding the Continuous Extension of Discrete Text: From Objective Mismatch to a Shift in Evaluation Emphasis: The paper argues that changing representation changes evaluation: latent reconstruction, prior fitting, and stabilization separate training from token-level likelihood, making generation quality and scaling behavior central capability measures.For Cola DLM, likelihood and perplexity are less directly aligned with the optimized objective than they are for models fitting discrete distributions.
8.3 Exploring Unified Models: Model-Environment Interaction and the Value of Multimodal Unification: Unified modeling is framed as learning in a non-separable multimodal environment, where joint observation, transition, and feedback regularities require more than superficially combining independent modalities.Continuous latent text provides an interface that can reduce the structural mismatch preventing discrete text from participating in shared multimodal interaction states.
8.4 The Three Themes Under a Unified Perspective: Across its three themes, the paper jointly changes the representational state, optimization objective, and learning environment, positioning hierarchical latent modeling as an alternative path toward unified representation, objectives, and environments.This perspective treats autoregressive modeling as one self-consistent design point rather than the unique organization of text generation.
A Strict Probabilistic Formulation and Training Objective Details of Cola DLM: The strict formulation defines Cola DLM as a hierarchical latent-variable model with a CNF-induced prior, an inference-only encoder, and a decoder-conditioned generative distribution; its ELBO is the likelihood-related training objective.Flow Matching efficiently learns the vector field for the same prior but is a practical prior solver, not the ELBO likelihood objective itself.
B.1 Sampling and Inference Procedure: Sampling is inherently two-stage: a base latent is transported backward to generate global semantics, then a decoder realizes text, with prefix-conditioned inference generating response latent blocks sequentially.The procedure supports autoregressive, non-autoregressive, or bidirectional conditional decoding after latent generation.
B.2 Unconditional Log-Likelihood and PPL Estimation: ELBO-based perplexity is an upper-bound-type estimate of true perplexity, while IWAE can provide tighter estimates; if the decoder supplies only a lower bound, exact perplexity is unavailable.These estimators apply to unconditional likelihood evaluation and inherit the distinction between the marginal likelihood and its variational bounds.
B.3 Conditional Probability Estimation and Block-Level Conditional Scoring: Conditional scoring subtracts prefix estimates from joint estimates using ELBO or IWAE plug-in formulas, but these conditional estimators do not automatically retain lower-bound guarantees.For block-causal priors, the same conditional calculation can be expressed blockwise over prefix and response latent blocks.

B.4 Numerical Evaluation of the CNF Prior and Estimation Algorithms … C.4 Difference from Plaid: Why Latent

The appendix specifies CNF-based likelihood estimators and places Cola DLM in a unified Markov-path framework, distinguishing its continuous latent prior and conditional realization from AR, LLaDA, and Plaid. The comparison identifies latent decomposition—not continuity alone—as the basis for separating semantic prior modeling from textual realization, subject to alignment with low-rate semantic structure.

B.4 Numerical Evaluation of the CNF Prior and Estimation Algorithms: The CNF prior supports unconditional ELBO/IWAE estimation by encoding posterior samples, integrating the augmented ODE, and recovering log pψ(z0) through the terminal base density and accumulated log-Jacobian.The procedure samples z0 from qϕ, solves the ODE to t = 1, computes the prior density, and combines it with decoder likelihood and encoder posterior terms.
B.4 Numerical Evaluation of the CNF Prior and Estimation Algorithms: Conditional log-likelihood is estimated as the joint full-sequence estimate minus the prefix estimate, using Algorithm A.1 as a black-box unconditional estimator.The method constructs xfull = (xpre, xres), estimates both log p(xfull) and log p(xpre), then subtracts them to obtain log p(xres | xpre).
C.1 A Unified Framework for Process-Based Generative Models: Cola DLM’s unified comparison distinguishes models by state space, trajectory, transition mechanism, path role, readout location, and objective tightness.The central distinction is whether the path represents observation-dependent corruption or an observation-independent latent prior.
C.2 Difference from Autoregressive Models: Why Diffusion: Unlike AR’s fixed prefix filtration and chain-rule factorization, Cola DLM uses latent marginalization, with continuous transport modeling pψ(z0) before conditional text realization.If the decoder is also non-autoregressive, generation first samples global semantic latents and then realizes text conditionally; the continuous prior improves fit to the aggregated posterior when expressive transport is beneficial.
C.3 Difference from Discrete Denoising Language Models: Why Continuous: LLaDA performs reverse recovery in discrete token space, whereas Cola DLM models a continuous latent prior with explicit log-density, Flow Matching geometry, interpolation, gradient guidance, and representation editing.These benefits arise from continuous-time change of variables and Euclidean vector-field regression, but gains depend on alignment between the latent structure and low-dimensional semantic organization.
C.4 Difference from Plaid: Why Latent: Compared with Plaid’s token-aligned, near-observation continuous representation, Cola DLM introduces an explicitly semantic latent variable that separates global representation from textual realization.When the encoder is nearly invertible, Cola DLM approaches Plaid’s non-compressive limit; its distinct contribution is the latent decomposition itself.
C.4 Difference from Plaid: Why Latent: The latent formulation decomposes representation learning into conditional reconstruction, mutual information Iq(X; Z0), and KL-based matching between the aggregated posterior and prior.This decomposition provides statistical benefits only when the data contain low-rate, high-value semantic structure, because the latent participates in marginalization rather than merely continuously relaxing tokens.

C.5 Why Cola DLM Is Not Merely a Latent Reverse-Diffusion LM … D.3 Structured Generation Regime: When Cola DLM Better Matches the True Mechanism

Cola DLM is a hierarchical latent-variable language model with a CNF prior, not merely a latent reverse-diffusion model. Its potential advantage depends on lower total statistical burden and on data whose global structure is compressible into informative latent factors.

C.5 Why Cola DLM Is Not Merely a Latent Reverse-Diffusion LM: Cola DLM transports a semantic prior from Gaussian noise, whereas Plaid and LLaDA reverse data-conditioned corruption processes.Its generation path is observation-independent prior transport rather than recovery of a particular sample x.
C.5 Why Cola DLM Is Not Merely a Latent Reverse-Diffusion LM: Cola DLM is first and foremost a hierarchical latent-variable language model with a CNF prior; flow or diffusion only makes that prior more expressive.The encoder belongs to variational inference and does not participate in generation, unlike the forward corruption process in Plaid and LLaDA.
D.1 A Unified Population-Level Comparison Criterion: All four model classes decompose population risk into data entropy, model mismatch, and an additional training-objective gap.This provides a unified basis for comparing AR, Cola DLM, Plaid, and LLaDA beyond model complexity alone.
D.1 A Unified Population-Level Comparison Criterion: At the population level, Cola DLM is better than a comparison class if and only if its total statistical burden is smaller.The criterion is expressed as R_Cola DLM < R_AR ⇐⇒ Cola DLM is better than AR at the population level.
D.2 Rate-Distortion and the Applicability Boundary of the Latent Bottleneck: A latent bottleneck helps when the latent retains enough information for reconstruction; low mutual information raises the best achievable reconstruction loss.The rate-distortion curve favors the bottleneck when low information rates achieve substantial reconstruction quality, but not when nearly every token detail is irreducible.
D.3 Structured Generation Regime: When Cola DLM Better Matches the True Mechanism: Cola DLM is expected to fit best when text is generated from low-dimensional, high-value global factors such as topic, plan, discourse structure, style, or constraints.In this structured regime, the model separates global semantic organization from conditional textual realization.
D.3 Structured Generation Regime: When Cola DLM Better Matches the True Mechanism: When inference and decoding gaps are small, Cola DLM’s burden is governed mainly by modeling the low-dimensional global factor rather than the full surface sequence.This decomposition is the stated source of its possible advantage over AR, LLaDA, and Plaid.

D.4 Three Curves, Theoretical Advantages, and Limitations

Cola DLM is theoretically advantageous only when its representation, prior-approximation, and inference-gap curves jointly favor latent decomposition. Its benefits over autoregressive and discrete diffusion models are offset by variational, bottleneck, optimization, and evaluation costs in unfavorable regimes.

Theoretical Limitations: Cola DLM may underperform when no low-rate sufficient representation exists, local surface form carries the semantics, or a strong token-space model is already adequate.In these regimes, costs include the bottleneck-induced reconstruction lower bound and the complexity of jointly training the encoder and other components.
Three Curves: Cola DLM succeeds only when the representation rate-distortion, prior-approximation, and inference-gap curves all favor latent decomposition.Otherwise, latent decomposition need not provide a theoretical advantage.
Theoretical Advantages: Compared with AR, Cola DLM can plan global semantics in latent space without a single left-to-right ordering, benefiting infilling, constraint satisfaction, and global planning.
Theoretical Advantages: Compared with discrete DLMs, Cola DLM exploits continuous latent geometry to support explicit log-densities and vector-field regression.
Theoretical Limitations: The latent bottleneck creates a variational gap and can cause posterior collapse when the decoder is too strong, the latent dimension too small, or regularization too strong.A poor encoder also incurs a systematic likelihood penalty by reducing Iq(X; Z0).
Theoretical Limitations: Evaluating log pψ(z0) can be more expensive than in AR models because it often requires ODE integration and divergence estimation.

E Existence of Global Semantic Structures … F.4 Why PPL and Generation Are Usually More Strongly Correlated in Discrete Models Such as AR / LLaDA

The analysis shows that latent-dimension-dependent shift drift rejects purely separable representations and supports shared structures affecting semantics. It also explains why Cola DLM can generate well despite inflated PPL: generation requires broad coverage, whereas PPL requires precise local probabilistic calibration.

E Existence of Global Semantic Structures: Stable, reproducible drift in the optimal timestep shift with latent dimension refutes purely local separability and implies cross-dimensional shared or joint latent structure.This conclusion does not uniquely establish a global semantic variable; if reconstruction remains stable while semantic metrics vary, the shared structure is implicated in high-level semantics.
E Existence of Global Semantic Structures: The shared-factor model explains shift drift mechanistically: increasing dimensions amplifies recoverable shared information, requiring recalibration of the diffusion shift to preserve an effective semantic-information regime.This explanatory model complements rather than replaces the rigorous contradiction argument, and VAE logSNR changes similarly alter effective semantic information through posterior uncertainty.
F.1.1 Flow Matching optimizes vector-field regression rather than gold-specific local log-density: Flow Matching learns a conditional mean velocity field through vector-field regression, not gold-specific local latent log-density calibration.For broad or multimodal conditional response distributions, this favors transporting probability mass into reasonable regions rather than maximizing density at one gold posterior tube.
F.1.2 Why the prior mean may be far from the ground truth while generation remains good: A prior mean can be far from a particular ground-truth latent while free-generation quality remains good when multimodal conditional mass lies within a decoder-good region.The posterior may select one narrow gold-specific mode even though the context-level prior represents several modes.
F.2.1 Generation quality and conditional PPL correspond to two different geometric objects: Good generation and conditional PPL measure different geometric requirements: generation needs prior mass in a decoder-good region, whereas PPL needs high local density on a specific gold posterior tube.Thus, broad coverage can coexist with a substantially biased conditional ELBO or plug-in PPL.
F.2.2 Why good reconstruction does not imply good conditional PPL: Good reconstruction does not imply good PPL because the conditional score also depends on the posterior–prior gap and their local probabilistic matching.A strong decoder reconstruction term can coexist with poor conditional scoring when the prior assigns insufficient density to the posterior.
F.3.1 Under a local Gaussian approximation, center alignment is necessary but not sufficient: Even when prior and posterior centers align, mismatched covariance scale, orientation, or volume can keep KL divergence and conditional PPL substantially poor.Continuous-density PPL therefore requires full local calibration rather than merely moving the prior mean near the ground-truth latent.
F.3.2 An overly sharp posterior further amplifies this issue: A highly concentrated posterior amplifies local density mismatches, so good PPL requires the prior to hit an especially narrow gold posterior tube.Consequently, continuous-latent PPL behaves more like local density calibration than a direct measure of generation quality; the supplied passages do not provide separate F.4 evidence.

F.4.1 AR: training, evaluation, and generation share the same discrete probabilistic object … G.1 Noise Schedule and logSNR as Equivalent Parameterizations

The section contrasts autoregressive and discrete diffusion objectives with Cola DLM’s latent-space integration and calibration requirements, then formalizes noise schedules and logSNR as equivalent descriptions of forward noising. It further explains how logSNR affects Flow Matching through sampling measure and target scaling, while timestep shifts recalibrate the effective semantic information regime.

F.4.1 AR: training, evaluation, and generation share the same discrete probabilistic object: Autoregressive training, perplexity evaluation, and generation operate on the same conditional probabilities, explaining their strong correlation with generation quality.The next-token cross-entropy minimizes the negative log of the same conditional terms queried during generation.
F.4.2 Discrete models optimize the probability mass of the correct event more directly: Discrete classifiers directly optimize probability mass on the correct event, whereas Cola DLM evaluates an integral over continuous latent space.The continuous-latent objective is additionally mediated by posterior–prior matching and decoder-emission compatibility.
F.4.2 Discrete models optimize the probability mass of the correct event more directly: Cola DLM’s conditional likelihood depends on local latent geometry, including mode location, volume, and curvature, rather than mode height alone.A Laplace approximation makes the contribution of local volume and curvature explicit.
F.4.3 LLaDA: the main discrepancy arises at the discrete observation level rather than the latent calibration level: Compared with Cola DLM, LLaDA’s main discrepancy remains concentrated in discrete token recovery rather than posterior–prior density collision in latent space.LLaDA still may exhibit a surrogate gap, but it recovers the discrete observed state without Cola DLM’s independent pointwise latent-prior matching requirement.
G.1 Noise Schedule and logSNR as Equivalent Parameterizations: The noise schedule and logSNR curve are equivalent parameterizations of the same forward noising geometry, with either one uniquely determining the signal and noise coefficients.They are not secondary transformations of the loss but alternative descriptions of the same process.
G A Unified Theoretical View of Noise Schedule, logSNR, and Flow Matching: In Flow Matching, logSNR changes the loss through both the sampling measure in the expectation and the scale of the supervised target velocity field.Thus, noise-schedule choices affect the training objective through two mathematically distinct pathways.
G A Unified Theoretical View of Noise Schedule, logSNR, and Flow Matching: Uniform-timestep training is generally not equivalent to uniform-logSNR training, so the chosen parameterization changes how training mass is distributed across noise regimes.The distinction follows from logSNR’s role in the sampling measure rather than from a mere relabeling of time.
G.1 Noise Schedule and logSNR as Equivalent Parameterizations: The optimal timestep shift remaps each raw timestep to a different logSNR regime, recalibrating training and sampling toward a more appropriate effective semantic information regime.Its role is therefore not merely to translate an abstract time label.

G.2 Reparameterization of the Flow Matching Objective … H Experiments Details

The appendix shows that logSNR reparameterization changes both Flow Matching’s sampling measure and target-velocity scale, while the noise schedule calibrates effective semantic information rather than raw timestep values. It further explains why latent dimension and VAE posterior uncertainty shift the optimum through their effects on semantic-information recovery.

G.2 Reparameterization of the Flow Matching Objective: Uniform-timestep training generally differs from uniform-logSNR training because the induced logSNR density is constant only when the schedule is affine.This follows from the change-of-variables relationship between timestep and logSNR sampling densities.
G.2 Reparameterization of the Flow Matching Objective: LogSNR affects Flow Matching through both the sampling weight across noise regimes and the numerical scale of the supervised target velocity field.Thus, the noise schedule directly determines which regimes receive emphasis and how difficult regression is within them.
G.3 Equivalent Derivation Under the Signal--Noise Parameterization: The same schedule-scaling conclusion holds under either an abstract bridge-path parameterization or the direct signal–noise coefficients (αt, σt).The appendix derives the relation from both parameterizations rather than treating it as specific to one notation.
G.4 The Optimal Schedule as Calibration of the Effective Semantic Information Regime: The schedule controls the effective semantic-information curve I(s; z_t), so timestep shifts recalibrate training and sampling toward a different semantic-information regime.The relevant object for the DiT is preserved semantic information about high-level variable s, not the raw timestep itself.
G.5 Why Latent Dimension and VAE logSNR Shift the Optimum: Increasing latent dimension can amplify recoverable shared semantic information, requiring a compensating timestep shift that naturally follows a logarithmic law.The shared-factor analysis attributes this effect to repeated observations of common low-dimensional semantic structure across latent dimensions.
G.5 Why Latent Dimension and VAE logSNR Shift the Optimum: Lower VAE posterior logSNR increases posterior variance and reduces effective semantic mutual information, shifting the schedule needed to maintain a comparable information regime.Latent dimension and posterior logSNR therefore act on the same semantic-information curve through aggregation and noise-floor effects, respectively.
G.5 Why Latent Dimension and VAE logSNR Shift the Optimum: Block size has no established closed-form schedule law, but it indirectly interacts with scheduling by changing semantic aggregation, cross-block dependence, and modeling difficulty.Its precise effect operates through the shape of the effective semantic-information curve rather than directly determining the schedule.

H.1 Dataset Introduction · H.2 Baseline Details

The evaluation spans eight benchmarks covering long-context continuation, knowledge, reasoning, comprehension, commonsense, and discourse-level coherence, with internal analyses using sampled subsets and external comparisons using broader test sets. Baselines comprise matched-from-scratch autoregressive and discrete diffusion models, enabling framework-focused comparisons.

H.1 Dataset Introduction: LAMBADA [74] tests whether models use distributed discourse context for difficult target-word prediction and maintain long-range contextual coherence.The benchmark emphasizes information beyond local context and serves as a representative continuation-style evaluation.
H.1 Dataset Introduction: MMLU [33], SIQA, and OBQA [64] assess broad factual knowledge, socially grounded commonsense, and multi-hop knowledge integration under constrained answer spaces.MMLU spans humanities, social sciences, STEM, and professional subjects; SIQA focuses on social situations, intentions, and reactions; OBQA combines science facts with commonsense knowledge.
H.1 Dataset Introduction: SQuAD [78] evaluates generative reading comprehension through accurate open-form answers conditioned on contextual evidence.It complements multiple-choice evaluations by testing open-form text generation.
H.1 Dataset Introduction: Story Cloze, RACE [48], and HellaSwag test narrative coherence, challenging passage inference, grounded commonsense, and plausible sentence continuation.These tasks evaluate discourse consistency, answer selection beyond span extraction, and discrimination between coherent continuations and distractors.
H.1 Dataset Introduction: The benchmark suite covers continuation, knowledge-intensive reasoning, reading comprehension, commonsense inference, and discourse-level semantic consistency across eight tasks.LAMBADA evaluates long-range contextual coherence; MMLU, SIQA, SQuAD, Story Cloze, OBQA, RACE, and HellaSwag cover complementary reasoning and generation abilities.
H.1 Dataset Introduction: Internal component analyses use randomly sampled test subsets from LAMBADA, MMLU, and SIQA, while external comparisons add five broader test-set evaluations.The added benchmarks are SQuAD, Story Cloze, OBQA, RACE, and HellaSwag, covering continuation, comprehension, knowledge-intensive reasoning, and commonsense.
H.2 Baseline Details: Internal analyses compare Cola DLM configurations without external baselines, whereas Section 4.5 uses external autoregressive and LLaDA baselines for scaling comparisons.The autoregressive model follows the official LLaMA framework [92], and LLaDA [70] is implemented with its official codebase; both are trained from scratch with random initialization.
H.2 Baseline Details: External comparisons train the autoregressive baseline, LLaDA [70], and Cola DLM independently from scratch with aligned corpus, parameter scale, and compute budget.This matching is intended to make reported differences primarily reflect the underlying modeling frameworks rather than training initialization or resource disparities.

H.3 Metrics and Evaluation · H.4 Setup Details · H.5 Why VAE Pretraining Does Not Compromise Fairness

The evaluation uses unified generative prompting and post-processing, while matched training and precision settings isolate modeling-paradigm differences. The fairness rationale treats VAE pretraining as representation stabilization rather than a hidden generative advantage, supported by stabilization and scaling evidence.

H.3 Metrics and Evaluation: All models use the same few-shot generative protocol with fixed templates and unified post-processing, while perplexity is not the primary metric because likelihood estimates can mismatch Cola DLM’s generation quality.Templates are shared across model families and evaluation examples, and default to 2-shot prompting unless otherwise noted.
H.3 Metrics and Evaluation: Multiple-choice outputs are truncated at the first newline, normalized, matched to candidate option texts, and counted correct only when the selected candidate equals the gold answer.Models generate option text rather than option labels; LAMBADA instead uses the first generated word, and SQuAD uses normalized exact-match short answers.
H.3 Metrics and Evaluation: Benchmark-specific formats preserve task semantics: MMLU, RACE, OBQA, HellaSwag, and SIQA use multiple-choice generation, Story Cloze uses an End: completion anchor, and LAMBADA uses open-ended continuation without demonstrations.SIQA presents three options, while the other listed multiple-choice tasks use four-way formats unless otherwise specified.
H.4 Setup Details: AR, LLaDA, and Cola DLM share corpus, tokenizer, sequence length, batch size, optimizer, learning-rate schedule, hardware, and evaluation protocol, with matched training tokens at matched optimization steps.AR and LLaDA use ∼1.8B non-embedding transformer parameters, while Cola DLM uses a comparable ∼1.8B DiT prior.
H.4 Setup Details: The precision regime is aligned: all models save and load checkpoints in fp32, keep runtime weights in fp32, use bfloat16 autocast for the main forward pass, and retain sensitive operations in fp32.These implementation-level checks are reported in Table 10.
H.5 Why VAE Pretraining Does Not Compromise Fairness: The additional VAE stage is framed as a method-specific representation-construction cost: it learns a stable text–latent interface, while generation remains the responsibility of the DiT prior and conditional decoder under matched settings.The pretrained encoder is used for variational inference and latent construction during training, not as the generative prior at inference.
H.5 Why VAE Pretraining Does Not Compromise Fairness: Figure 15 shows that AR representations stabilize after an early phase, while the pretrained VAE encoder exhibits an even more explicit stabilization pattern.The evidence concerns inter-step drift and distance to the first checkpoint, which become small or saturate after early training.
H.5 Why VAE Pretraining Does Not Compromise Fairness: VAE pretraining does not itself explain Cola DLM’s gains: Fix VAE saturates earlier, whereas Joint DiT x1 shows the strongest overall scaling trend.This pattern indicates that subsequent prior learning and co-adaptation, rather than a pretrained encoder alone, drive the observed advantage.

H.6 Fixed vs. Evolving Latent Space Additional Results … I Discussion

Additional results show that continuously co-evolving the latent space with DiT outperforms interval updates even after pretrained VAE initialization, while the appendices define VAE logSNR and explain how timestep-shift parameters alter the semantic regimes emphasized during training.

H.6 Fixed vs. Evolving Latent Space Additional Results: Implementation checks compare AR, LLaDA, and Cola DLM under closely aligned checkpoint precision, runtime autocast policy, and numerically sensitive operations.These checks address numerical comparability across the three methods.
H.7 VAE LogSNR Formula: VAE logSNR is defined as the log ratio between signal power in the posterior mean and noise power in the posterior variance across latent coordinates.The expectation covers the data distribution and latent dimensions, and the quantity is estimated empirically over samples and latent coordinates.
H.7 VAE LogSNR Formula: A larger VAE logSNR indicates a cleaner, more deterministic latent representation because posterior-mean signal dominates posterior uncertainty.Throughout the paper, VAE logSNR refers specifically to the quantity defined in Eq. (H.2).
H.8 Noise Schedule Timestep Shift Visualization: LogitNormal timestep schedules shift training emphasis across the denoising trajectory, with µ moving sampling toward later timesteps and σ controlling concentration or dispersion.This changes which semantic-information or logSNR regime is emphasized rather than merely tuning an isolated numerical hyperparameter.
H.9 Timestep Shift Formula: Timestep shift samples a normalized timestep from a LogitNormal distribution, maps it to (0, 1) with the logistic function, and rescales it to the diffusion interval.The resulting continuous timestep is converted to a valid discrete training timestep by rounding or flooring.
H.6 Fixed vs. Evolving Latent Space Additional Results: Joint DiT x1 achieves stronger overall trends and better final performance than Interval across Task Average, LAMBADA, MMLU, and SIQA despite pretrained VAE initialization.This isolates continuous versus intermittent latent updates after stable initialization and supports sustained latent-space co-evolution with DiT.
H.9 Timestep Shift Formula: Under the timestep-shift parameterization, larger µ moves probability toward later timesteps, smaller σ sharpens concentration, and larger σ disperses sampling.After rescaling, the schedule directly changes the distribution of sampled logSNR regimes during training.
I Discussion: The timestep-shift visualization compares uniform and LogitNormal schedules to show how µ and σ alter the logSNR regime emphasized during training.Increasing µ shifts sampling mass later, while changing σ controls concentration sharpness.

I.1 Understanding the Conditioning Strategies in the First Generation Block … Text to Text

Clean condition repaint best preserves the known region while transporting only the unknown region in the first generation block. Qualitative results also show coherent text continuation, text-to-image synthesis, and image–text-to-text generation from a unified model trained with limited data.

I.1 Understanding the Conditioning Strategies in the First Generation Block: Partial repaint weakens conditioning by replacing the clean known region with a noisy, time-varying surrogate, increasing irreducible uncertainty and accumulating trajectory error.Reducing m worsens performance, while increasing t from 1 to 3 provides no stable gains because early corrections cannot remove later structural mismatch.
I.1 Understanding the Conditioning Strategies in the First Generation Block: Clean condition repaint consistently performs best across all three tasks, while padding generally exceeds partial repaint but remains below explicit clean conditioning.The first block should generate zU under fixed (zpre, zK), rather than jointly re-estimating known and unknown regions.
I.2 More Unified Approach Results with Image Modalities: A unified model trained jointly across modalities produces coherent text, image, and image–text outputs despite limited training data.The model is pretrained from scratch on multiple task types with adjusted data proportions across training stages.
I.2.1 Text to Text: Text-to-text samples are generally coherent and fluent after training on approximately 1B text tokens.Figure 18 spans conversational, narrative, expository, technical, and article-style continuation examples.
Text to Text: Additional text-to-text examples demonstrate coherent continuation across heterogeneous styles, including conversational, narrative, expository, technical, and article-style text.These qualitative examples support the unified latent-prior formulation’s ability to handle varied textual continuation settings.