PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Continuous Diffusion Language Models

Hui Ren, Zihan Li, Chang Liu, Huidong Liu, Alexander Schwing

arXiv:2610.02193v1cs.CLcs.AIcs.LG

TL;DR

Parallel discrete diffusion decodes tokens independently, while continuous diffusion does not tie its denoising trajectory to tokens until final decoding. HC-DLM uses a persistent continuous latent with per-step token readouts that feed back into latent denoising. It improves over discrete and continuous diffusion baselines at matched model size across structured reasoning, mathematical planning, and language modeling.

  • Problem

    Parallel discrete decoding models simultaneously decoded tokens as independent marginals, while continuous diffusion does not involve tokens along its trajectory until final decoding.

    Source: §1 · p. 1

  • Method

    HC-DLM uses a continuous latent as its only persistent generative state, reading out tokens at each reverse step and feeding them back to condition the next latent update.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    HC-DLM improves over discrete and continuous diffusion baselines at matched model size across Sudoku, Countdown, and LM1B.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The reported evaluations show the framework applies across structured reasoning, mathematical planning, and unconditional language modeling.

    Source: §6 · p. 10

    Training each step is more expensive than a purely discrete masked diffusion baseline, and experiments target moderate-scale benchmarks; scaling to larger pretrained backbones is limited by compute resources.

    Source: D LIMITATIONS · pp. 23–24

Abstract

from arXiv · show

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

1 INTRODUCTION

HC-DLM combines a persistent continuous latent with token readouts that feed back into denoising, targeting the token-dependence gap in parallel decoding and the weak token anchoring of continuous diffusion. The paper reports improvements over discrete and continuous diffusion baselines at matched model size across Sudoku, Countdown, and LM1B.

1 INTRODUCTION

  • Parallel discrete decoding models simultaneously decoded tokens as independent marginals, missing dependencies among tokens constrained by syntax, logic, or physical rules.The supplied discussion notes that discrete diffusion can still support bidirectional conditioning and iterative refinement.Source: §1 · p. 1
  • Continuous diffusion shares a latent across tokens, but its denoiser uses only that state and does not involve tokens along the trajectory until final decoding.The introduction contrasts this with hybrid methods that use separate transition chains.Source: §1 · pp. 1–2
  • HC-DLM makes the continuous latent its only persistent state, reading tokens from it at each reverse step and feeding them back to condition the next latent transition.The token state has its own forward kernel, distinguishing it from a fed-back estimate; the paper calls this organization hierarchical coupling.Source: §1 · p. 2
  • HC-DLM improves over discrete and continuous diffusion baselines at matched model size on Sudoku, Countdown, and LM1B.The reported evaluations cover puzzle accuracy on Sudoku and Countdown and generative perplexity on LM1B.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries describe discrete diffusion as corruption and reversal of token sequences, and continuous diffusion as denoising or transporting continuous data from noise. They also identify independent token sampling as a limitation of parallel discrete decoding.

2 PRELIMINARIES

  • Discrete diffusion progressively corrupts a token sequence toward a stationary distribution, with absorbing-mask and uniform corruption as common choices.A token predictor reverses the corruption; when multiple positions are decoded simultaneously, their joint is modeled as a product of marginals.Source: §2 · pp. 2–3
  • Parallel discrete prediction samples each token independently, so the joint distribution over simultaneously decoded tokens does not capture their statistical dependencies.Source: §2 · p. 3
  • Continuous DDPMs use a Gaussian forward process and weighted MSE denoising, while Flow Matching trains a velocity field to transport noise toward data.Flow Matching samples can be generated by integrating the learned velocity backward from t = 1 to t = 0.Source: §2 · p. 3

3 HIERARCHICAL CONTINUOUS DIFFUSION LANGUAGE MODELS

HC-DLM pairs independently corrupted continuous and discrete trajectories with a reverse hierarchy in which tokens are read from the latent and condition its next update. A variational lower bound supplies the training objective, and inference alternates latent denoising, token readout, and token re-noising.

3.1 GENERATIVE MODEL AND FORWARD PROCESS

  • HC-DLM introduces a coupled continuous–discrete trajectory whose reverse process reads tokens from the latent and feeds them back as a scaffold for latent updates.The latent is the persistent state; the token state has no transition chain of its own and is read out anew at each step.Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 VARIATIONAL LOWER BOUND

  • Independent continuous and discrete forward corruptions permit a variational lower bound over the coupled reverse process, with reconstruction, prior-matching, boundary, and denoising terms.The bound is stated as a lower bound on log p(k0).Source: §3.2 · pp. 4–5

3.3 PRACTICAL TRAINING OBJECTIVE

  • The practical objective combines token reconstruction, encoder entropy, and token-conditioned continuous denoising signals.The denoising term is implemented with Conditional Flow Matching; the encoder entropy term discourages a deterministic encoder map.Source: §3.3 · pp. 5–6, §3.2 · p. 5

3.4 INFERENCE

  • Sampling starts from Gaussian latent noise and a stationary token state, then alternates scaffold-conditioned latent denoising, token readout, and token re-noising.After the final step, the sampler returns the token prediction from x0.Source: §3.4 · p. 6, §3.3 · p. 6

4 EXPERIMENTS

Across Sudoku, Countdown, and LM1B, HC-DLM is evaluated against autoregressive, discrete-diffusion, and continuous or hybrid diffusion baselines. Results favor HC-DLM over matched hybrid baselines on the structured tasks and show the best generative perplexity among evaluated diffusion models on LM1B; ablations examine its components and generation dynamics.

4 EXPERIMENTS

  • The experiments compare HC-DLM with autoregressive, discrete-diffusion, and continuous or hybrid baselines across Sudoku, Countdown, and LM1B.The matched CCDD baseline uses a 6M-parameter scale and the same protocol on Sudoku and Countdown; parameter counts refer to sampling-time generative parameters, excluding token embeddings.Source: §4 · p. 6, §4.1 · p. 7

4.2 TOKEN-DEPENDENT REASONING: SUDOKU

  • On Sudoku, HC-DLM leads matched CCDD on the Hard split, 72.41 vs. 70.73, while CCDD is marginally higher on Easy, 94.65 vs. 94.21.HC-DLM also surpasses the best masked-diffusion variant and a much larger autoregressive model on Easy.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 MATHEMATICAL REASONING: COUNTDOWN

  • On Countdown, HC-DLM surpasses matched hybrid CCDD on both subtasks, with the CD5 margin at 37.52 vs. 25.35, while much larger discrete diffusion models achieve the best overall numbers.The authors interpret the broader gap between diffusion variants and larger autoregressive models as consistent with any-order decoding suiting subgoal-imbalanced planning, not as a property of one method.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 LANGUAGE MODELING: LM1B

  • On LM1B, HC-DLM achieves the best generative perplexity among evaluated diffusion models, outperforming discrete diffusion and continuous models that decode only once at the end.Entropy remains stable under moderate reductions in NFE and across the CFG sweep; at large batch sizes, HC-DLM also maintains lower wall-clock sampling time.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 ABLATION STUDIES

  • Removing either the continuous latent trajectory or discrete conditioning degrades Sudoku performance, especially on the Hard split.The ablation associates a latent without a discrete scaffold with over-smoothing precise token-level constraints, and independent discrete sampling with a bottleneck.Source: §4.5 · pp. 8–9, Table 4 · p. 8
  • The uniform discrete kernel is substantially stronger than the absorbing kernel for the hierarchical sampler, while adaptive ordering modestly improves the absorbing kernel.The authors attribute the gap to uniform corruption preserving an informative scaffold and per-step readout keeping every position revisable.Source: §4.5 · p. 9, Table 5 · p. 9
  • HC-DLM’s intermediate Sudoku accuracy rises earlier and reaches a higher plateau than Latent DM, while its final accuracy remains robust as denoising steps are reduced.The authors describe the discrete scaffold as helping structural constraints lock in sooner through the feedback loop; MDM degrades more sharply with fewer steps.Source: §4.5 · p. 9

5 RELATED WORK

Prior work includes discrete diffusion over tokens, continuous diffusion over sequence states, and hybrid methods pairing discrete and continuous channels. HC-DLM is distinguished by reading tokens from the latent at each step rather than maintaining a separate token transition chain.

Hybrid discrete–continuous diffusion

  • Figure 3 contrasts HC-DLM’s faster intermediate Sudoku accuracy gains with its robustness to fewer denoising steps.The figure’s second panel compares final Hard Sudoku accuracy across step counts.Source: Figure 3 · p. 9

Discrete diffusion language models

  • Discrete diffusion methods span absorbing- and uniform-state token diffusion, alternative decompositions, and reported advantages on planning tasks.HC-DLM preserves parallel token updates while adding a continuous variable.Source: §5 · pp. 9–10

Continuous diffusion for text

  • Continuous text-diffusion models denoise per-token embeddings or encoder-compressed latents, then decode tokens from the clean result.HC-DLM instead reads tokens from the latent at every step and feeds them back into its denoiser.Source: §5 · p. 10

Hybrid discrete–continuous diffusion

  • Hybrid methods pair a discrete chain with a global latent, per-token hints, or a parallel embedding chain, while retaining a token transition chain of their own.HC-DLM reads tokens from the latent at each step, so its tokens have no separate transition kernel.Source: §5 · p. 10

6 CONCLUSION

HC-DLM couples token readouts and a continuous latent trajectory in a two-level reverse process. The conclusion attributes its structured-task results to this coupling and the scaffold feedback.

6 CONCLUSION

  • HC-DLM uses a two-level reverse chain in which tokens are read from a continuous latent at every step and fed back to its denoiser.The continuous latent is the only state that persists across steps.Source: §6 · p. 10
  • A single variational lower bound supplies the coupled training objective, routing cross-token dependence through a compact learned latent.Source: §6 · p. 10
  • Experiments span Sudoku, Countdown, and LM1B; structured-task comparisons show scaffold feedback matters and the coupled model outperforms continuous-only and discrete alternatives.Source: §6 · p. 10

A PROOF OF PROPOSITION 1

This appendix derives the variational bound and its learning signals, then describes the practical model components and conditional-generation extension.

A PROOF OF PROPOSITION 1

  • The proof applies Jensen’s inequality to the token-sequence marginal likelihood and combines the resulting terms into Proposition 1’s bound.The terms include reconstruction, prior matching, boundary, and denoising contributions.Source: A PROOF OF PROPOSITION 1 · pp. 13–14
  • The joint KL splits into discrete token prediction and continuous denoising signals because the forward latent posterior is independent of the current token.Source: A PROOF OF PROPOSITION 1 · p. 14
  • A shared clean-token predictor avoids a separate head for every noise level, and its clean-token cross-entropy upper-bounds the trainable noisy-token KL.The data-processing inequality supplies the bound; the absorbing kernel can yield the usual masked-diffusion form under the stated non-mask support condition.Source: A PROOF OF PROPOSITION 1 · pp. 15–16

B.1 ARCHITECTURE

  • The practical architecture jointly trains an encoder, token predictor, and denoiser; the denoiser conditions on both noisy latents and tokens, while the predictor reads clean latents.The noisy-token law is formed by marginalizing the clean-token distribution through the known categorical kernel.Source: §B.1 · pp. 16–17

B.2 CONDITIONAL GENERATION

  • For conditional generation, the bound conditions each learned component on the requested condition while leaving the forward noising kernels unchanged.The condition is supplied as an additional embedding, requiring no other architectural changes.Source: §B.2 · p. 17

B.3 DATA AND SERIALIZATION DETAILS

The appendix specifies puzzle splits, correctness criteria, and sequence formats for Sudoku and Countdown.

B.3 DATA AND SERIALIZATION DETAILS

  • Sudoku uses 81-token grids with a unique solution, and a prediction is correct only when all 81 cells match it.The standard split contains puzzles solvable by seven fixed logical strategies; the hard split contains the remaining puzzles requiring another strategy.Source: §B.3 · p. 17
  • Countdown targets range from 10 to 100; CD4 and CD5 use four or five inputs, with CD5 requiring longer chains and a larger search space.Source: §B.3 · p. 17
  • A Countdown answer is correct when its full arithmetic chain obeys the operation, number-use, intermediate-result, exact-division, and target constraints.Agreement with the dataset’s particular solution is not required; examples are serialized as a prompt and comma-separated sub-equations.Source: §B.3 · p. 18

B.4 TRAINING DETAILS

The appendix reports model configurations, optimization settings, and compute context for structured tasks and LM1B.

B.4 TRAINING DETAILS

  • Structured-task models use one latent token and 50 Euler sampling steps, with the denoiser and token predictor comprising the reported sampling-time parameters.The encoder is training-only; the modules have approximately 8M, 4.9M, and 1.7M parameters, respectively.Source: §B.4 · p. 18
  • Structured-task training uses Adam at 1×10−4, batch size 256 per GPU across seven L40 GPUs, and a BERT-style reconstruction loss weighted 0.2.Configurations were repeated over multiple independent runs; EMA used decay 0.999.Source: §B.4 · p. 18
  • The LM1B model has 118M reported sampling-time parameters and is evaluated on mean GPT-2-Large perplexity from 1,024 samples generated with 128 steps.Training uses two stages totaling 660K steps and took approximately 48 hours on one RTX PRO 6000 GPU.Source: §B.4 · p. 18
  • Validation perplexity is not reported because likelihood evaluation for flow-based models can require additional likelihood-specific training.Source: §B.4 · p. 18
  • Reported LM1B training times are compute context, not a hardware-controlled efficiency comparison, because hardware and schedules differ across runs.The cited durations are end-to-end costs rather than direct throughput measurements.Source: §B.4 · p. 19

B.5 SAMPLE QUALITY

Across LM1B sampling tests, HC-DLM guidance lowers generative perplexity while preserving entropy, and sampling is faster at larger batch sizes. The section also describes token-channel guidance and reports quality metrics across sampling budgets.

B.5 SAMPLE QUALITY

  • Table 7 reports generative perplexity and entropy across NFE budgets, computed over 1,024 unconditional samples at each budget.Lower generative perplexity and higher entropy are preferred.Source: §B.5 · p. 19, Table 7 · p. 19

B.7 CLASSIFIER-FREE GUIDANCE

  • Token-channel CFG trains the denoiser with scaffold-conditioned and uninformative channels, then uses w > 1 to increase reliance on the token scaffold.The uninformative channel replaces the full row with independently uniform random token IDs.Source: §B.7 · pp. 19–20

B.6 INFERENCE SPEED

  • At batch sizes 16 and above, HC-DLM has faster LM1B sampling than the comparison method, with the gap widening as batches grow.Figure 4 measures wall-clock sampling time per batch at 128 NFEs and sequence length 128; lower is better.Source: Figure 4 · p. 20

B.7 CLASSIFIER-FREE GUIDANCE

  • Guidance lowers generative perplexity while preserving entropy across the CFG sweep, and the main LM1B result uses w = 2.75.Figure 5 compares generative perplexity and entropy as w varies; w = 1 is sampling without guidance extrapolation.Source: §B.7 · p. 20, Figure 5 · p. 20

B.8 LM1B GENERATION SAMPLES

The appendices provide LM1B samples and probe latent capacity and inference time, then explain the uniform-kernel rationale and compare HC-DLM with VMD, CADD, and CCDD.

B.8 LM1B GENERATION SAMPLES

  • Table 8 provides three unconditional LM1B generation examples from runs using 128 NFEs.The examples are presented as samples, not as a separate quantitative evaluation.Source: §B.8 · p. 20, Table 8 · p. 21

B.9 LATENT SEQUENCE LENGTH

  • Across M = 1, 2, 4, 8, and 16, Sudoku accuracy suggests gains up to M = 4 before degradation, while larger M costs additional compute.M controls continuous-channel capacity by setting the number of continuous tokens available to the denoiser.Source: §B.9 · p. 21, Table 9 · p. 21

B.10 WALL-CLOCK INFERENCE TIME

  • At 100 denoising steps, HC-DLM takes roughly 2.3 seconds per batch versus about 3.3 seconds for MDM, despite using an additional continuous state.The comparison is at the same number of steps; the section says this implementation improves solution quality without increasing measured sampling latency.Source: §B.10 · pp. 21–22, Figure 6 · p. 22

B.11 WHY THE UNIFORM KERNEL SUITS THE HIERARCHY

  • Uniform corruption gives every position a concrete token that remains revisable at each step, unlike absorbing corruption, which freezes commitments as positions unmask.The paper argues that revisability suits constraint-dense tasks, where one early error can invalidate a solution.Source: §B.11 · p. 22

C.1 VMD

  • VMD draws one global Gaussian latent and holds it fixed, whereas HC-DLM refines its latent trajectory at every step under token-scaffold feedback.In VMD, the discrete chain remains the generative process, and its static context cannot adapt as the partially decoded sequence changes.Source: §C.1 · p. 22, Table 10 · p. 23

C.2 CADD

  • CADD’s hints recycle its own token predictions, leave unmasking to discrete diffusion, and require absorbing corruption rather than uniform corruption.HC-DLM instead uses a learned latent denoiser, with tokens read out from the latent.Source: §C.2 · p. 23

C.3 CCDD

  • CCDD independently corrupts tokens and frozen contextual embeddings, then updates both channels through separate heads and standard rules.HC-DLM has no separate token head; its tokens come from the latent alone.Source: §C.3 · p. 23

C.4 SUMMARY

  • HC-DLM rereads tokens from the latent at every step instead of carrying decoded tokens forward, and the uniform kernel works better than the absorbing kernel in the experiments.This distinguishes its token-generation structure from the three compared hybrid models.Source: §C.4 · p. 23

D LIMITATIONS

The limitations discussion notes higher training cost, moderate-scale experimental scope, and compute constraints on evaluating larger pretrained backbones.

D LIMITATIONS

  • Jointly training the encoder, continuous denoiser, and token predictor makes each training step more expensive than in purely discrete masked diffusion.The encoder is discarded at sampling time, and inference is reported as faster than the baseline at matched step counts.Source: D LIMITATIONS · p. 23
  • Concatenation-based conditioning on the token scaffold is simple and effective, but more structured fusion may improve sample efficiency.Source: D LIMITATIONS · pp. 23–24
  • Experiments focus on structured reasoning and planning at moderate scale, where correctness can be verified exactly and controlled comparisons fit an academic compute budget.The paper identifies scaling to larger pretrained backbones as future work constrained by compute resources typically available in academia.Source: D LIMITATIONS · p. 24

Related papers

Loading 2610.02193v1…