Continuous Latent Diffusion Language Model

• Existing approaches struggle to jointly provide efficient generation, scalable representation, and global semantic modeling across autoregressive and diffusion paradigms.Autoregressive models incur sequential inference and fixed-order bias, discrete diffusion uses costly multi-step observation recovery, and continuous diffusion usually models token-aligned representations rather than latent priors.Source: §1 · pp. 1–2
• Cola DLM uses a Text VAE, block-causal DiT, and conditional decoder to model latent priors before realizing local text.Diffusion performs latent prior transport rather than token-level observation recovery, while block causality preserves cross-block structure with parallel computation within blocks.Source: §1 · p. 2
• The framework decomposes generation into global semantic modeling and local textual realization within a unified probabilistic hierarchy.This weakens fixed token-order inductive bias, supports semantic compression and prior fitting in continuous space, and provides a modular path toward other continuous modalities.Source: §1 · p. 2
• A unified Markov-path analysis characterizes Cola DLM’s advantages in global semantic modeling, non-autoregressive inductive bias, and theoretical interpretability.The analysis frames diffusion as latent prior transport and separates semantic organization from textual realization.Source: §1 · p. 2
• Experiments span 4 research questions, 8 benchmarks, matched ∼2B-parameter autoregressive and LLaDA baselines, and scaling curves reaching about 2000 EFLOPs.They identify an effective overall configuration and validate strong potential and favorable scaling behavior for text generation.Source: §1 · p. 2
• Additional analyses examine likelihood–generation mismatch, first-block conditioning, latent compression, and extension from discrete text toward continuous modalities such as vision.These studies point toward a broader unified generative paradigm beyond the core framework.Source: §1 · p. 2

2.1 Autoregressive Language Models

• Autoregressive models [56] [92] [101] impose fixed left-to-right, sequential generation, whereas Cola DLM uses global latent semantics and block-causal conditional decoding.This targets limitations in infilling, local editing, global reorganization, and inference efficiency.Source: §2.1 · p. 3

2.2 Discrete Diffusion Language Models

• Discrete diffusion models use transition kernels [2] [10] [88] or masking states [69] [70] [80] [81] [84] [105] [113] [114] [117] [118], limiting sampling speed, semantic planning, or control.Cola DLM moves diffusion into compressible continuous latents carrying global semantics.Source: §2.2 · p. 3

2.3 Continuous Diffusion Language Models

• Continuous diffusion methods use vocabulary-aligned representations [31] [43] [59] [79], token embeddings [14] [21] [27] [29] [52] [54] [87], or latent spaces [44] [58] [63] [109], but lack Cola DLM’s hierarchical latent-variable framework.Cola DLM separates global semantics from local realization and learns a semantic prior in a dynamic continuous latent space.Source: §2.3 · p. 3

3.1.1 Theoretical Formulation of Cola DLM

• Cola DLM factorizes generation into a continuous latent prior pψ(z0) for global semantics and a conditional decoder pθ(x | z0) for discrete text realization.The encoder qϕ(z0 | x) is used for variational inference during training, not as part of the generative model; Flow Matching implements prior transport rather than defining the model.Source: §3.1.1 · p. 4

3.1.2 Probability Estimation for Cola DLM

• Cola DLM estimates unconditional likelihoods with ELBO- or IWAE-style estimators and obtains conditional probabilities by subtracting prefix likelihood from joint-sequence likelihood.The IWAE-style estimator is typically tighter, while the conditional estimator scores the joint sequence and prefix with the same unconditional procedure.Source: §3.1.2 · p. 5

3.2 Workflow of Cola DLM

• Cola DLM trains a causal Text VAE first, then jointly learns a block-causal DiT prior, and finally generates latent response blocks before conditional decoding.The VAE uses reconstruction, KL, and masking losses; Stage 2 controls gradients and regularizes latent drift while preserving the autoencoding structure.Source: Figure 1 · p. 6, §3.2.1 · p. 6, §3.2.2 · p. 6

3.2.2 Prior Learning with Block-Causal DiT

• The block-causal DiT attends bidirectionally within each latent block but causally across blocks, enabling block-wise prior learning conditioned on historical clean latent blocks.At inference, the prefix is encoded into clean latent conditions, response blocks are generated by transporting noise seeds, and the decoder realizes the response text.Source: §3.2.2 · p. 6, §3.2.3 · p. 7

3.3.1 Text Modeling under a Unified Stochastic-Path View

• Under a unified Markov-path view, AR follows left-to-right prefix expansion, LLaDA performs discrete observation recovery, Plaid performs continuous token-aligned recovery, whereas Cola DLM transports only a latent prior.Cola DLM therefore uses continuity in pψ(z0), separating global semantic compression from local token realization instead of treating diffusion as text observation recovery.Source: §3.3.1 · pp. 7–8

3.3.2 Theoretical Advantages of Cola DLM

• Cola DLM is advantageous when its total statistical burden, including variational inference, prior approximation, and reconstruction terms, is lower than a comparison model’s burden.A latent bottleneck helps when data has low-dimensional global semantics and high-dimensional local realization; diffusion or continuity alone does not guarantee benefit.Source: §3.3.2 · p. 9

4.1 Experimental Setup

• Experiments use matched roughly 2B-parameter models, unified few-shot accuracy evaluation, and internal benchmarks LAMBADA [74], MMLU [33], and SIQA.External comparisons additionally use SQuAD [78], Story Cloze, OBQA [64], RACE [48], and HellaSwag, while perplexity is not used as the primary comparison metric.Source: §4.1 · p. 10

4.2 Evidence of Global Semantic Structures in Cola DLM (RQ1)

• Optimal timeshift shifts from approximately 1.0 at d = 16 to 1.7 at d = 64 and 2.3 at d = 128, consistently across semantic metrics, supporting shared global latent structures.The empirical peaks also broadly match theoretical predictions, contradicting a fully separable latent-space explanation.Source: §4.2 · pp. 10–11, Figure 2 · p. 11

4.3 Analysis of Different Latent Spaces in Cola DLM (RQ2)

• Joint DiT with a stable pretrained VAE achieves the strongest scaling and final performance, whereas fixed, scratch-trained, weakly updated, or intermittently frozen latent spaces underperform.Continuous and sufficiently strong latent updates are beneficial only when anchored by meaningful initialization.Source: §4.3 · pp. 12–13, Figure 3 · p. 12
• The average score rises from 8.7 at d = 16 to 11.3 at d = 64 and 11.8 at d = 128, while higher dimensions partially reduce collapse and shift the best timeshift toward larger loc values.BERT loss improves performance when the VAE learning-rate ratio is 1, while learnable VAE logSNR delivers the strongest overall performance under the current setup.Source: §4.3 · pp. 13–15

4.4 Ablation on the Diffusion Process in Cola DLM (RQ3)

• The diffusion ablations jointly show that both DiT block granularity and noise-schedule calibration materially affect downstream generation quality under joint latent evolution.The training-stage study varies block size and schedule location under a pretrained, jointly optimized VAE–DiT configuration.Source: §4.4 · p. 16, §4.4.1 · p. 17

4.4.1 Training Stage

• Block size 16 gives the best overall performance at both checkpoints, while larger blocks degrade results and block size 1 remains competitive but weaker.The findings indicate that moderate local grouping balances local processing and semantic aggregation better than either fully fine-grained or overly coarse partitioning.Source: §4.4.1 · pp. 16–17, Figure 6 · p. 16
• loc = 1.0 achieves the highest Task Average at both the 30K and 40K checkpoints and the best or near-best task results, indicating calibrated semantic-information recovery.Changing schedule location shifts the effective logSNR trajectory, so the preferred schedule depends on alignment between denoising and latent semantic scale.Source: §4.4.1 · pp. 17–18, Figure 7 · p. 17, Figure 8 · p. 18
• Joint DiT performs best when the denoising logSNR schedule uses loc=1, matching or surpassing Fix VAE baselines, whereas loc=0 and uniform schedules remain weaker.The result indicates that joint latent evolution requires a schedule aligned with the semantic structure of the evolving latent space.Source: §4.4.1 · p. 18
• Noise-schedule sensitivity reflects calibration of the semantic variable’s effective mutual-information curve, linking schedule location, latent dimension, and VAE logSNR.For the current Joint DiT setting with d = 16, loc=1.0 supports stable optimization and better alignment between denoising and the latent semantic-information regime.Source: §4.4.1 · p. 18

4.4.2 Inference Stage

• Increasing denoising steps yields large early gains, but performance saturates after roughly 16–32 steps; 8–10 steps recover most final performance and reduce sequential depth by 1.6–2.0× versus AR.Under Joint DiT with jointly optimized pretrained VAE and DiT, moderate inference budgets around 10–32 steps provide the strongest efficiency–accuracy trade-off.Source: §4.4.2 · p. 19
• CFG has a non-monotonic effect: Task Average is best around CFG 3–6, while guidance beyond approximately 10 degrades performance and very large values severely distort denoising.Thus, moderate guidance improves conditional semantic recovery, whereas excessive guidance harms the denoising trajectory.Source: §4.4.2 · pp. 19–20

4.5 Comparison of Scaling Performance (RQ4)

• Cola DLM steadily improves across compute budgets and reaches the best final Task Average, outperforming matched AR and LLaDA baselines in the high-compute regime.Figure 10 evaluates matched 1.8B-parameter backbones under a unified few-shot generative protocol; the result is conservative because larger latent dimensions and improved logSNR settings remain unexplored.Source: Figure 10 · p. 20, §4.5 · pp. 20–21
• Cola DLM shows strong scaling on MMLU, RACE, Story Cloze, and OBQA, while eventually surpassing AR on SQuAD and remaining close to AR on LAMBADA.These gains are strongest at medium-to-large compute budgets and remain informative despite lower absolute multiple-choice scores under generative evaluation.Source: §4.5 · p. 21

5.1 The Structural Gap Between Likelihood-Oriented Estimation and Generation Quality

• Good generation and good likelihood-oriented estimation are not equivalent: generation needs semantically valid latent regions, while PPL also requires local calibration near the gold posterior.Figure 11 and Table 4 support this distinction: for “at,” PPL falls from 1.15 × 10^6 to 641.57 and then 245.36 while outputs deteriorate from “on” to “in” and then a comma.Source: Figure 11 · p. 22, §5.1 · p. 23, Table 4 · p. 23

5.2 Impact of Conditioning and Padding Strategies in the First Generation Block

• Clean condition repaint performs best in the first mixed generation block, outperforming partial repaint and left/right padding because persistent conditioning preserves the known prompt region.Reducing the guided fraction generally hurts performance, while increasing repaint repetitions from t = 1 to t = 3 provides no stable gains.Source: Table 5 · p. 24, §5.2 · p. 25

5.3 Compression of the Latent Space

• Patch size 2 is much worse overall than patch size 1, but becomes competitive and can slightly surpass it when prompt lengths are divisible by 2.The weakness mainly comes from non-divisible boundaries: biased compressed prompt latents propagate through denoising, whereas aligned compression can improve semantic abstraction and generation efficiency by covering 32 rather than 16 text tokens per block.Source: §5.3 · pp. 25–26, Table 6 · p. 25

5.4 Robustness of VAE Latent Reconstruction

• The Text VAE reconstructs nearly perfectly at t = 0 and retains approximately 0.92 accuracy at t = 250, indicating graceful degradation under increasing noise.These results support the VAE latent space as a stable semantic interface rather than a fragile compressed code.Source: §5.4 · p. 26, Figure 13 · p. 26

5.5 Towards a Unified Approach with Image Modalities

• Cola DLM naturally extends to unified text–image modeling by mapping modalities into continuous latents, using a shared prior for global and cross-modal semantics, and modality-specific decoders for realization.The prototype supports text continuation, image-conditioned text generation, and text-to-image generation through a shared block-causal MMDiT prior.Source: §5.5 · p. 26, MMDiT · p. 27

MMDiT

• The unified text–image prototype provides qualitative feasibility evidence rather than a mature multimodal result because it uses early-stage moderate pretraining on in-house 256- and 640-resolution data without extensive curation or supervised fine-tuning.The authors position the formulation as a scalable bridge toward native unified generative models and plan more comprehensive multimodal training.Source: Figure 14 · p. 27, MMDiT · p. 28

Scale and evaluation

• Current experiments provide initial feasibility, competitiveness, and scaling evidence but remain relatively controlled, motivating evaluation with larger models, longer training, and greater compute.The paper frames these experiments as clarifying the framework’s key properties rather than establishing its upper bound.Source: §6 · p. 28

Model design

• Text VAE training, compression, latent dimensionality, semantic smoothness, and joint VAE logSNR, DiT block size, and noise-schedule calibration all affect latent organization and generation quality.Stronger latent representations generally appear to require better-aligned noise calibration, leaving substantial room for optimization.Source: §6 · p. 28

Framework extensions

• Cola DLM’s central opportunity is to separate global semantic prior modeling from local textual realization, enabling stronger latent modules such as AE and RAE [112].The paper emphasizes that this decomposition, rather than denoising alone, is the framework’s main value.Source: §6 · p. 28
• More flexible prior-learning methods, including drifting-model-based [19] distribution matching for continuous priors, could further develop the framework.This direction follows directly from treating diffusion as continuous-prior modeling rather than only denoising.Source: §6 · p. 28

Unified generation

• The unified continuous latent-space perspective may extend Cola DLM beyond text to continuous modalities such as images, supporting broader unified generation.This prospect is proposed as a longer-term direction rather than a demonstrated result in the current study.Source: §6 · p. 28

7 Conclusion

• Cola DLM decomposes text generation into global semantic prior modeling in latent space and local textual realization through conditional decoding.This hierarchical information decomposition is presented as a principled alternative to strictly token-level language modeling.Source: §7 · p. 28
• Strictly matched experiments show strong generation quality and encouraging scaling behavior for Cola DLM.The conclusion attributes these results to the combined theoretical analysis and experimental study.Source: §7 · p. 28
• The study finds shared global semantic structure in latent space and identifies effective choices for latent-space formation and diffusion modeling.Source: §7 · p. 28
• Generation-oriented evaluation and scaling trends may be more informative than likelihood alone for this class of models.Source: §7 · p. 28
• The continuous latent formulation offers a concrete path toward unified modeling across discrete text and continuous modalities.Source: §7 · p. 28

8 Afterword: Research Objectives and Significance

• The afterword formalizes learning as model–environment interaction in which representation, feedback objectives, and environmental structure jointly determine what the model learns.The environment includes data distributions, task formats, supervision protocols, and rules converting feedback into optimization signals.Source: §8 · p. 29

8.1 Rethinking Text Modeling Paradigms: From State Space in the System to Hierarchical Text Generation

• Cola DLM organizes global semantics in a continuous latent state before conditionally realizing local text, rather than treating token-level autoregression as the unique design.Its latent formulation separates conditional realization, information compression, and prior matching.Source: §8.1 · p. 30

8.2 Understanding the Continuous Extension of Discrete Text: From Objective Mismatch to a Shift in Evaluation Emphasis

• For hierarchical continuous latent models, generation quality and scaling behavior can align better with learned capability than perplexity because training jointly targets reconstruction, latent prior fitting, and representation stabilization.Applicability additionally depends on favorable rate–distortion behavior, decreasing latent-prior approximation error, and a controllable inference gap.Source: §8.2 · p. 31

8.3 Exploring Unified Models: Model-Environment Interaction and the Value of Multimodal Unification

• Unified models matter because multimodal environments contain joint observation, transition, and feedback regularities that cannot generally be learned by independently modeling each modality.Cola DLM offers discrete text a continuous latent interface, reducing its representational mismatch with continuous modalities and enabling participation in shared interaction states.Source: §8.3 · pp. 32–33

8.4 The Three Themes Under a Unified Perspective

• The paper unifies its contribution around three coupled changes: hierarchical latent representation, objective evaluation beyond direct likelihood, and continuous interfaces for richer multimodal environments.Together, these changes position Cola DLM as both an alternative text-generation route and a framework for jointly considering representation, objective alignment, and environment design.Source: §8.4 · p. 33

A Strict Probabilistic Formulation and Training Objective Details of Cola DLM

• The appendix gives a strict probabilistic formulation of Cola DLM, covering its latent-variable model, CNF prior, marginal likelihood, ELBO, and Flow Matching’s theoretical role.It defines the training components needed to connect the model’s hierarchical generation process with its optimization objective.Source: A Strict Probabilistic Formulation and… · p. 41

A.1 Probabilistic Objects, CNF Prior, and Generative Distribution

• Cola DLM is fundamentally a hierarchical latent-variable language model whose generative distribution combines a continuous latent prior with a conditional text decoder, rather than a reverse process directly defined in token space.The variational posterior is used only during training, while the generative distribution is defined by pθ(x | z0) and pψ(z0).Source: §A.1 · pp. 41–42

A.2 Explicit Density of the CNF Prior and the Outer ELBO

• The CNF prior has an explicit log-density and deterministic Markov-semigroup structure arising from prior transport, after which variational inference handles the latent integral in the marginal likelihood.This Markov structure is not an observation-dependent corruption–recovery mechanism.Source: §A.2 · p. 42
• The outer ELBO is the variational lower bound optimized during training, distinct from the true marginal log-likelihood log p(x).When the decoder computes log pθ(x | z0), this defines Cola DLM’s strict training objective.Source: §A.2 · pp. 42–43

A.3 Average ELBO, Aggregated Posterior, and Prior Matching

• With encoder and decoder fixed, prior learning fits pψ to the aggregated posterior, while encoder mutual information controls compression and the division between latent semantics and decoder realization.The encoder determines the prior’s target distribution ¯qϕ(z0) and the retained information Iq(X; Z0).Source: §A.3 · p. 43

A.4 The Role of Flow Matching: A Prior Solver Rather Than the Model Definition

• Flow Matching efficiently learns a vector field transporting the aggregated posterior to the base distribution, but it is a prior solver rather than the probabilistic likelihood objective.Its loss cannot be identified term-by-term with −log pψ(z0) in the ELBO.Source: §A.4 · pp. 43–44

A.5 Strict Form of the Training Objective

• The strict training objective is maximization of the ELBO, or equivalently minimization of its negative; Flow Matching is only an efficient implementation for learning the prior.Thus, the lower-bound interpretation belongs to the ELBO rather than the Flow Matching loss.Source: §A.5 · p. 44

B.1 Sampling and Inference Procedure

• Sampling is inherently two-stage: backward integration generates global latent semantics from a base seed, and a conditional decoder realizes the text.For continuation, the prefix is encoded first, response latents are generated block by block, and the response is then decoded conditionally.Source: §B.1 · p. 45

B.2 Unconditional Log-Likelihood and PPL Estimation

• ELBO-based perplexity is an upper-bound-type estimate of true perplexity, while IWAE generally provides a tighter estimate when the decoder can compute log pθ(x | z0).If the decoder supplies only a lower bound, evaluation can report only bound-based perplexity rather than exact perplexity.Source: §B.2 · p. 45

B.3 Conditional Probability Estimation and Block-Level Conditional Scoring

• Conditional scoring can subtract prefix estimates from joint estimates, but these plug-in estimators do not automatically inherit lower-bound guarantees and require careful treatment of prefix-latent uncertainty.Strict conditional probability integrates over both response latents and posterior uncertainty in the prefix latent; blockwise forms support candidate ranking and multiple-choice scoring.Source: §B.3 · p. 46

B.4 Numerical Evaluation of the CNF Prior and Estimation Algorithms

• The CNF estimators compute unconditional likelihoods by sampling encoder latents, integrating an augmented ODE to evaluate the prior density, and combining decoder, prior, and posterior terms.Algorithm A.1 supports both ELBO-style and IWAE-style estimation, while the divergence can use Hutchinson’s trace estimator with a fixed random vector per ODE trajectory.Source: §B.4 · p. 47
• Conditional response likelihoods are estimated by subtracting the prefix estimate from the joint full-sequence estimate, while a direct local score can rank candidate response blocks.Algorithm A.2 treats Algorithm A.1 as a black box and applies the conditional identity d log p(xres | xpre) = d log p(xfull) − d log p(xpre).Source: §B.4 · pp. 47–48

C.1 A Unified Framework for Process-Based Generative Models

• The unified framework distinguishes AR, LLaDA, Plaid, and Cola DLM by state space, trajectory, transitions, path role, observation readout, and objective tightness.The key distinction is whether the path is an observation-dependent corruption path or an observation-independent latent prior path.Source: §C.1 · p. 48

C.2 Difference from Autoregressive Models: Why Diffusion

• Unlike AR’s fixed prefix filtration, Cola DLM uses latent marginalization, with continuous prior transport and optionally non-autoregressive decoding removing dependence on a prescribed token order.Its diffusion-like component improves fitting of the continuous prior pψ(z0) to the aggregated posterior rather than avoiding maximum likelihood.Source: §C.2 · p. 49

C.3 Difference from Discrete Denoising Language Models: Why Continuous

• Compared with discrete token-space denoising, Cola DLM’s continuous latent formulation supplies explicit change-of-variables log densities, Flow Matching for prior geometry, and support for interpolation, guidance, and editing.These advantages depend on continuous structure aligning with low-dimensional semantic organization rather than following from continuity alone.Source: §C.3 · p. 50

C.4 Difference from Plaid: Why Latent

• Plaid remains close to token-aligned observations, whereas Cola DLM’s genuinely new ingredient is a potentially compressive latent decomposition between semantic representation and textual realization.When the encoder is nearly invertible, Cola DLM approaches Plaid’s near-observation limit.Source: §C.4 · p. 50
• Cola DLM introduces an explicit latent variable that separates global semantic planning from local textual realization through prior modeling and conditional decoding.Its ELBO decomposes into reconstruction, retained mutual information Iq(X; Z0), and aggregated-posterior-to-prior matching terms; this benefits generation when data contain low-rate, high-value semantic structure.Source: §C.4 · p. 50

C.5 Why Cola DLM Is Not Merely a Latent Reverse-Diffusion LM

• Cola DLM transports a semantic prior from Gaussian latent space, whereas Plaid and LLaDA corrupt specific observations before reversal.Its generation path is observation-independent, making it prior transport rather than data corruption.Source: §C.5 · p. 51
• Cola DLM is fundamentally a hierarchical latent-variable language model with a CNF prior, while flow or diffusion only makes that prior more expressive.The encoder belongs to variational inference and does not participate in generation, unlike the corruption processes defining Plaid and LLaDA.Source: §C.5 · p. 51

D Theoretical Advantages, Applicability, and Limitations

• The theoretical comparison asks when Cola DLM is better or worse than AR, LLaDA, and Plaid, rather than assuming greater complexity is beneficial.The framework evaluates models under a unified theoretical criterion.Source: D Theoretical Advantages,… · p. 51

D.1 A Unified Population-Level Comparison Criterion

• All four model classes decompose population risk into data entropy, model mismatch, and an additional training-objective gap, so superiority depends on total statistical burden.Cola DLM includes an inference gap, while Plaid and LLaDA have reverse/path-posterior and surrogate gaps, respectively.Source: §D.1 · p. 52

D.2 Rate-Distortion and the Applicability Boundary of the Latent Bottleneck

• A latent bottleneck helps when modest mutual information preserves high-value text structure, but it raises reconstruction loss when the representation discards essential information.The rate-distortion curve favors Cola DLM when global topics, plans, style, and constraints are compressible, but not when nearly every token detail is irreducible.Source: §D.2 · p. 53

D.3 Structured Generation Regime: When Cola DLM Better Matches the True Mechanism

• Cola DLM is especially well matched when low-dimensional global factors determine topic, planning, discourse structure, style, or other constraints that shape text realization.This is the favorable regime of low-dimensional but high-value latent factors.Source: §D.3 · p. 53
• In that regime, the encoder recovers global information, the prior models its distribution, and the decoder realizes text conditionally, reducing the burden to modeling the global factor.This possible advantage over AR, LLaDA, and Plaid requires small inference error, accurate latent-prior fitting, and effective conditional realization.Source: §D.3 · p. 53

Theoretical Limitations

• When no low-rate sufficient representation exists, local form carries the semantics, or token-space modeling is already adequate, Cola DLM may underperform and incur bottleneck and joint-training costs.The bottleneck raises H(X | Z0), while jointly training the encoder and Cola DLM adds optimization complexity.Source: §D.4 · pp. 53–54

Three Curves

• Cola DLM is theoretically advantageous only when its representation rate-distortion, prior-approximation, and inference-gap curves all favor latent decomposition.Success or failure is framed as competition among D(R), the prior-approximation curve, and G_infer.Source: §D.4 · p. 54

Theoretical Advantages

• Compared with AR, discrete DLMs, and Plaid, Cola DLM enables global latent planning, continuous-space operations, and explicit semantic compression under hierarchical data structure.Its ELBO decomposition also separates representation, prior, and decoder analysis.Source: §D.4 · p. 54

Theoretical Limitations

• Cola DLM has a variational gap, and poor encoding, posterior collapse, costly ODE-based likelihood evaluation, or absent low-dimensional semantics can erase its advantage.Collapse may occur when the decoder is too strong, latent dimensionality is too small, or regularization is too strong; evaluation can require ODE integration and divergence estimation.Source: §D.4 · p. 54

E Existence of Global Semantic Structures

• Systematic, reproducible drift of the optimal timestep shift with latent dimension refutes the purely local and separable null hypothesis.Under the null, changing dimension only rescales the objective and cannot move its optimal shift; the rigorous conclusion is shared or joint cross-dimensional structure, not necessarily a global semantic factor.Source: E Existence of Global Semantic… · p. 55
• Changing latent dimension or VAE logSNR shifts the effective semantic mutual-information regime, so both can require timestep-shift recalibration.Dimension changes aggregate and amplify shared information across channels, whereas lower VAE logSNR raises posterior uncertainty and lowers effective semantic SNR at a fixed timestep.Source: E Existence of Global Semantic… · pp. 56–57
• When reconstruction remains stable but semantic metrics vary with shift, the drift indicates shared latent structures that affect high-level semantic organization.This pattern distinguishes recalibration of shared semantic information from changes in local token realization, strengthening the interpretation beyond mere cross-dimensional dependence.Source: E Existence of Global Semantic… · p. 58

F Theoretical Analysis of Likelihood Estimation Bias

• The likelihood analysis distinguishes exact conditional marginal probability from a local estimator involving prior, decoder, and posterior-density terms.The gap between these objects is the probabilistic basis for analyzing why local conditional scores can remain inflated despite reasonable free-generation quality.Source: F Theoretical Analysis of Likelihood… · pp. 58–59

F.1.1 Flow Matching optimizes vector-field regression rather than gold-specific local log-density

• Flow Matching fits a time-dependent vector field through dynamical regression rather than directly maximizing local conditional log-density at a gold latent point.Its squared-loss pointwise optimum is a conditional mean velocity field, not a mode-seeking or gold-specific velocity.Source: §F.1.1 · p. 59

Then the conditional risk

• The conditional-risk minimizer is f⋆(z, t, c) = E[Y | Zt = z, t, c].The result follows by decomposing prediction error around the conditional mean, with the cross term vanishing under conditional expectation.Source: Then the conditional risk · p. 59
• Flow Matching learns conditional mean dynamics and may favor an average transport map over gold-specific local density calibration when conditional responses are multimodal or broad-peaked.This bias moves probability mass into a reasonable region without guaranteeing local calibration around a particular sample.Source: Then the conditional risk · p. 59

F.1.2 Why the prior mean may be far from the ground truth while generation remains good

• A prior mean can be far from a ground-truth latent yet preserve good free-generation quality when multimodal conditional mass lies within a decoder-good region.The discrepancy may reflect a multimodal context-level distribution whose posterior selects one narrower gold-specific mode.Source: §F.1.2 · p. 60

F.2.1 Generation quality and conditional PPL correspond to two different geometric objects

• Generation quality requires prior mass to enter a decoder-good region, whereas conditional PPL additionally requires high local density along a particular gold posterior tube.Thus, good generation and poor conditional PPL can coexist because coverage and calibration are different requirements.Source: §F.2.1 · pp. 60–61

F.2.2 Why good reconstruction does not imply good PPL

• Good reconstruction near the ground truth does not guarantee good PPL when the posterior–prior gap remains large, because PPL also depends on local posterior–prior matching.Reconstruction and conditional density calibration are therefore distinct objectives in continuous latent language models.Source: §F.2.2 · pp. 61–62

F.3 Even If the Prior Mean Is Close to the Ground Truth, PPL May Still Be Poor

• Even with approximately aligned prior and posterior centers, local covariance, orientation, curvature, and volume terms can still make PPL poor.The broader issue is that continuous latent PPL depends on detailed local geometry, not only positional agreement.Source: §F.3 · p. 62

F.3.1 Under a local Gaussian approximation, center alignment is necessary but not sufficient

• Even when the prior mean is close to the ground-truth latent, mismatched covariance scale, orientation, or volume can keep KL divergence and conditional PPL poor.Continuous-density PPL therefore requires local probabilistic calibration beyond center alignment.Source: §F.3.1 · p. 62

F.3.2 An overly sharp posterior further amplifies this issue

• A highly concentrated posterior amplifies local log-density gaps, so good PPL requires the prior to hit a much narrower gold posterior tube.In this regime, PPL behaves more like local density calibration than a direct generation-quality metric.Source: §F.3.2 · p. 62

F.4.1 AR: training, evaluation, and generation share the same discrete probabilistic object

• In autoregressive models, training, perplexity evaluation, and generation operate on the same conditional probabilities, explaining their strong correlation.The next-token cross-entropy is the negative log of the probabilities queried during generation and evaluated by PPL.Source: §F.4.1 · p. 63

F.4.2 Discrete models optimize the probability mass of the correct event more directly

• Discrete models directly optimize the probability mass of the correct event, whereas continuous latent likelihood additionally depends on latent-space integration and local geometry.Continuous latent models also require posterior–prior matching and decoder-emission compatibility.Source: §F.4.2 · p. 63

F.4.3 LLaDA: the main discrepancy arises at the discrete observation level rather than the latent calibration level

• LLaDA’s discrepancy remains concentrated in discrete token recovery, so its perplexity–generation correlation is generally stronger than in continuous latent models.Unlike Cola DLM, LLaDA does not require matching a sample-dependent continuous posterior with an independent conditional prior.Source: §F.4.3 · p. 63

G A Unified Theoretical View of Noise Schedule, logSNR, and Flow Matching

• The unified theory identifies noise schedules and logSNR curves as equivalent parameterizations of the same forward noising process.It treats the schedule as controlling the mathematical object entering training and semantic modeling.Source: G A Unified Theoretical View of Noise… · p. 64, §G.1 · p. 64
• In Flow Matching, logSNR affects loss through both the sampling measure and the scale of the supervised target velocity field.These two pathways jointly determine how noise-level weighting enters the training objective.Source: G A Unified Theoretical View of Noise… · p. 64
• Uniform-timestep training is generally not equivalent to uniform-logSNR training, so schedule choices alter the effective weighting of semantic regimes.The distinction follows from logSNR’s role in both expectation sampling and target scaling.Source: G A Unified Theoretical View of Noise… · p. 64

G.1 Noise Schedule and logSNR as Equivalent Parameterizations

• An optimal timestep shift remaps each raw timestep to a different logSNR regime, recalibrating training and sampling toward a more appropriate semantic information regime.The shift changes the signal and noise coefficients rather than merely translating an abstract timestep label.Source: G A Unified Theoretical View of Noise… · p. 64, §G.1 · p. 64

G.2 Reparameterization of the Flow Matching Objective

• Uniform-timestep training generally differs from uniform-logSNR training because the induced logSNR density is constant only when the schedule is affine.The change of variables introduces a density determined by the schedule derivative, so non-affine schedules emphasize logSNR regimes unevenly.Source: §G.2 · p. 65
• The logSNR parameterization changes Flow Matching through two mechanisms: it reweights sampled noise regimes and rescales the supervised target velocity field.Thus, the noise schedule directly determines both which logSNR regions receive training emphasis and the regression difficulty within those regions.Source: §G.2 · p. 66

G.3 Equivalent Derivation Under the Signal--Noise Parameterization

• The same schedule-dependent scaling arises from the signal–noise coefficients (αt, σt), confirming that the conclusion is independent of whether the derivation starts from bridge paths or diffusion parameters.The appendix derives the relation directly from the signal–noise parameterization and obtains the same result.Source: §G.3 · pp. 66–67

G.4 The Optimal Schedule as Calibration of the Effective Semantic Information Regime

• The schedule should be interpreted as calibrating the effective semantic-information curve I(s; zt), not as tuning an isolated numerical timestep hyperparameter.Timestep shifts recalibrate training and sampling toward different semantic-information regimes, with the optimum depending on latent dimension, uncertainty, geometry, block size, and model parameters.Source: §G.4 · p. 67

G.5 Why Latent Dimension and VAE logSNR Shift the Optimum

• Increasing latent dimension strengthens recoverable shared semantic information, so maintaining a comparable recovery regime naturally requires a logarithmically varying timestep shift.The shared-factor model explains this through aggregation of repeated observations across latent dimensions.Source: §G.5 · p. 68
• Lower VAE posterior logSNR increases posterior uncertainty and reduces effective semantic mutual information, requiring a corresponding schedule adjustment.Latent dimension and posterior logSNR therefore act on the same effective semantic-information curve, respectively through shared-information aggregation and the noise floor.Source: §G.5 · p. 68
• Block size has no established closed-form schedule law, but it couples to scheduling through within-block aggregation, cross-block dependence, and the geometry of the block-causal prior.The precise claim is that block size indirectly changes the effective semantic-information curve rather than directly determining the schedule.Source: §G.5 · p. 68

H.1 Dataset Introduction

• The benchmark suite covers long-context continuation, factual knowledge, commonsense reasoning, reading comprehension, narrative coherence, and discourse-level semantic consistency.It includes LAMBADA [74], MMLU [33], SIQA, SQuAD [78], Story Cloze, and OBQA [64].Source: §H.1 · p. 69
• LAMBADA [74] tests whether models use distributed discourse context for difficult word prediction and maintain long-range contextual coherence.The target word is generally difficult to infer from local context alone.Source: §H.1 · p. 69
• MMLU [33], SIQA, and OBQA [64] assess knowledge-intensive answering, socially grounded commonsense inference, and integration of science facts with broader commonsense knowledge.These tasks use multiple-choice settings and may require reasoning beyond direct fact recall.Source: §H.1 · p. 69
• SQuAD [78], RACE [48], Story Cloze, and HellaSwag evaluate contextual answer generation, reading comprehension, narrative coherence, and plausible continuation selection.Together, they test open-form generation, passage inference, causal reasoning, and semantic consistency.Source: §H.1 · p. 69
• Internal component analyses use randomly sampled LAMBADA, MMLU, and SIQA test subsets, whereas Section 4.5 evaluates all eight test sets for broader external comparison.The external suite adds SQuAD, Story Cloze, OBQA, RACE, and HellaSwag to assess complementary task types.Source: §H.1 · p. 69

H.2 Baseline Details

• The autoregressive baseline follows the LLaMA framework [92], uses random initialization and next-token prediction, and matches Cola DLM’s data, tokenizer, budget, scale, and optimization settings where applicable.It serves as the primary strong decoder-only autoregressive reference.Source: §H.2 · p. 70
• LLaDA [70] provides a discrete diffusion baseline using iterative denoising in token space, with matched data, tokenizer, compute budget, and model size in scaling experiments.Internal analyses compare Cola DLM configurations only, while external baselines are mainly used for Section 4.5 scaling comparisons.Source: §H.2 · p. 70
• For external comparisons, all models are trained from scratch with aligned corpora, parameter scales, and compute budgets, so differences primarily reflect modeling frameworks.This excludes pretrained initialization and resource mismatches as intended sources of advantage.Source: §H.2 · p. 70

H.3 Metrics and Evaluation

• All models use unified few-shot generative evaluation with shared demonstrations, fixed templates, answer generation, post-processing, and matching instead of perplexity as the primary comparison metric.Unless otherwise noted, prompts use 2-shot demonstrations, and multiple-choice models generate option text rather than labels.Source: §H.3 · p. 70, Table 7 · p. 71
• Scoring truncates multiple-choice outputs at the first newline and matches normalized text to candidate answers, while LAMBADA uses the first continuation word and SQuAD uses normalized short-answer exact match.Templates, demonstrations, truncation, and matching rules are identical across AR, LLaDA, and Cola DLM; only native generation differs.Source: §H.3 · pp. 70, 72, 74

H.4 Setup Details

• AR, LLaDA, and Cola DLM share the corpus, tokenizer family, sequence length, batch size, optimizer, learning-rate schedule, hardware, and evaluation protocol, with comparable ∼1.8B generation backbones.Cola DLM’s remaining parameters come from the additional VAE required for continuous latent modeling, so matched optimization steps expose each method to the same training-token budget.Source: §H.4 · pp. 74–75, Table 8 · p. 75
• The methods also use aligned numerical regimes: fp32 checkpointing and runtime weights, bfloat16 autocast for main computation, and fp32 for normalization and softmax.The additional VAE encode/decode path is intrinsic to continuous latent modeling rather than an extra numerical advantage.Source: §H.4 · p. 74

H.5 Why VAE Pretraining Does Not Compromise Fairness

• Figure 15 shows that AR representations and the pretrained VAE encoder stabilize early, after which the dominant learning burden shifts to modeling generation dynamics.Both models exhibit rapidly decreasing inter-step drift and later saturation relative to early checkpoints, with the VAE encoder showing especially explicit stabilization.Source: §H.5 · p. 75
• VAE pretraining mainly amortizes representation stabilization, while generation remains dependent on the DiT prior and conditional decoder rather than a pretrained text generator.The VAE encoder supports latent construction during training, whereas inference generates new content through the DiT prior before decoding.Source: §H.5 · pp. 75–76
• Fix VAE saturates earlier while Joint DiT x1 has the strongest overall scaling trend, indicating that subsequent prior learning and DiT co-adaptation drive gains beyond a frozen pretrained encoder.The comparison suggests that a pretrained encoder alone cannot explain the final performance pattern.Source: §H.5 · p. 76

H.7 VAE LogSNR Formula

• VAE logSNR is defined as the log ratio between posterior-mean signal power and posterior-variance noise power across latent coordinates.The quantity is evaluated over the data distribution and latent dimensions, with empirical estimation using sample and latent-coordinate averages.Source: §H.7 · p. 77
• A larger VAE logSNR indicates a cleaner, more deterministic latent representation because posterior-mean signal dominates posterior uncertainty.Throughout the paper, VAE logSNR refers specifically to the quantity defined in Eq. (H.2).Source: §H.7 · p. 77

H.8 Noise Schedule Timestep Shift Visualization

• Representation-space drift stabilizes early: AR hidden-state geometry quickly reaches a stable regime, while pretrained VAE encoding remains nearly flat during DiT training.This indicates that later generation performance depends primarily on backbone modeling rather than continued embedding or latent-space formation.Source: Figure 15 · p. 78

H.9 Timestep Shift Formula

• Timestep shift samples a normalized timestep from a LogitNormal distribution, maps it through the logistic function to (0, 1), and rescales it to t ∈ (0, T).Discrete training timesteps are obtained by rounding or flooring the continuous timestep to the valid grid.Source: §H.9 · p. 78

H.6 Fixed vs. Evolving Latent Space Additional Results

• Joint DiT x1 consistently outperforms Interval across Task Average, LAMBADA, MMLU, and SIQA even with pretrained VAE initialization.This isolates continuous versus intermittent latent updates after stable initialization and supports sustained co-evolution between the latent space and DiT prior.Source: §H.6 · pp. 76–77, Figure 16 · p. 79

H.9 Timestep Shift Formula

• Under the timestep-shift parameterization, larger µ moves probability toward later timesteps, whereas smaller σ sharpens concentration and larger σ disperses sampling.The resulting schedule directly changes the sampled logSNR regimes during training.Source: §H.9 · p. 79

H.8 Noise Schedule Timestep Shift Visualization

• LogitNormal timestep schedules shift training emphasis across denoising regimes: increasing µ favors later timesteps, while σ controls concentration versus dispersion.Thus, timestep shift changes which semantic-information regime is sampled more frequently rather than merely tuning an isolated numerical hyperparameter.Source: §H.8 · pp. 77–78, Figure 17 · p. 80

I Discussion

• Figure 17 summarizes that timestep shift changes the logSNR regime emphasized during training by varying the location and sharpness of LogitNormal sampling mass.It compares uniform sampling with distributions under different µ and σ settings.Source: Figure 17 · p. 80

I.1 Understanding the Conditioning Strategies in the First Generation Block

• The first generation block must preserve its known region as a stable boundary condition while generating the unknown region, rather than jointly re-estimating both parts.This mixed block differs from later purely generative blocks because known and unknown positions coexist and interact within it.Source: §I.1 · pp. 80–81
• Partial repaint increases irreducible conditional-velocity uncertainty by replacing the clean known region with a noisy, time-varying surrogate during only part of the trajectory.Its condition-role mismatch also causes trajectory bias to accumulate, while reducing m worsens performance and increasing t cannot remove the later-stage mismatch.Source: §I.1 · p. 83
• Clean condition repaint consistently performs best across all three tasks, while shorter partial-repaint guidance hurts and increasing repaint cycles from t = 1 to t = 3 yields no stable gains.The strategy matches the intended conditional Flow Matching formulation: preserve the known region continuously and generate only the unknown region.Source: §I.1 · p. 84
• Left and right padding are often stronger than partial repaint but remain below clean condition repaint because layout changes do not explicitly lock the known region during denoising.Padding can also complicate the block-causal attention structure by reshaping dependencies between known and unknown regions.Source: §I.1 · p. 84

I.2 More Unified Approach Results with Image Modalities

• A single unified model trained jointly from scratch on multiple task types produces promising cross-modal text and image samples despite limited data.The training data proportions were adjusted across stages, and the reported results suggest potential for multimodal understanding and generation.Source: §I.2 · p. 84

I.2.1 Text to Text

• Text-to-text generations are generally coherent and fluent despite training on only approximately 1B text tokens.The additional examples span conversational, narrative, expository, technical, and article-style continuation.Source: §I.2.1 · p. 84, Figure 18 · p. 86

I.2.3 Image × Text to Text

• After training with merely approximately 5M image–text pairs, image-and-text-to-text outputs are generally fluent and semantically aligned, showing preliminary captioning ability.The images in these examples were generated by external image-generation models from ground-truth captions.Source: §I.2.3 · p. 85

Text to Text

• Figure 29 shows additional image×text-to-text examples involving people-centric, object-centric, and illustration-style inputs.Source: Figure 29 · p. 97
• Figure 30 presents additional qualitative results for apparel, accessories, vehicles, architecture, and scenic human portraits.Source: Figure 30 · p. 98
• Figure 31 provides further examples covering sports, apparel, portraits, and stadium scenes, including caption-generation prompts.Source: Figure 31 · p. 99