PaperlayerView exact revision on arXiv

Source-linked AI summary

Next Semantic Scale Prediction via Hierarchical Diffusion Language Models

Cai Zhou, Chenyu Wang, Dinghuai Zhang, Shangyuan Tong, Yifei Wang, Stephen Bates, Tommi Jaakkola

arXiv:2510.08632v1cs.CLcs.LG

TL;DR

Masked and uniform discrete diffusion have limitations in noisy-token semantics and self-correction. HDLM introduces hierarchical semantic scales for progressive refinement and a flexible diffusion framework. The authors report that small HDLM models outperform other discrete diffusion variants on validation and generative perplexity.

  • Problem

    Masked diffusion gives all masked tokens the same embedding and cannot revise decoded tokens, while uniform diffusion introduces semantically inconsistent noise and is empirically difficult to denoise.

    Source: §1 · p. 1

  • Method

    HDLM maps fine-grained tokens to coarser semantic hierarchies in the forward process and progressively predicts more detailed tokens in reverse, with a closed-form diffusion ELBO.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Small HDLM models outperform other discrete diffusion variants on both validation and generative perplexity.

    Source: §4.2 · p. 9

  • Takeaways & Limitations

    The authors conclude that HDLM demonstrates strong language-modeling capability in their extensive experiments.

    Source: §6 · p. 10

    Experimental implementations of multiple intermediate hierarchy levels are left for future work, and they may require more careful sampling-algorithm design.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

In this paper we introduce Hierarchical Diffusion Language Models (HDLM) -- a novel family of discrete diffusion models for language modeling. HDLM builds on a hierarchical vocabulary where low-level tokens with detailed semantics are surjectively mapped to high-level tokens with coarse-grained meanings. In the forward process, each token is independently perturbed to its higher-level ancestor with more abstract semantics according to the scheduler, while in the reverse process the model progressively predicts the next, more detailed semantics. Taken together, HDLM provides a general time-varying next semantic scale prediction process for language modeling. We derive closed-form expressions for the diffusion Evidence Lower Bound (ELBO), and show that HDLM can be implemented in a flexible manner while including the existing MDLM as a special case. We also propose practical training techniques based on the insights. Extensive text generation experiments validate the effectiveness of HDLM, which demonstrates consistently lower validation and generative perplexity than baselines.

1 Introduction

Autoregressive models cannot revise generated tokens, while masked and uniform discrete diffusion each have semantic or correction limitations. HDLM introduces hierarchical semantic scales for progressive token refinement and derives a flexible framework with a closed-form training ELBO.

1 Introduction

  • Autoregressive generation cannot revise earlier tokens, while masked diffusion uses identical mask embeddings and uniform diffusion introduces semantically inconsistent random tokens.Uniform diffusion also denoises from a more complex distribution and empirically performs poorly; GIDD retains limitations in noisy-token semantics and correction.Source: §1 · pp. 1–2
  • HDLM progressively maps tokens to more abstract higher-level ancestors, then predicts increasingly detailed tokens through time-varying next-scale prediction.Tokens in a sentence may occupy different hierarchies at the same timestep, and the hierarchy can include multiple intermediate vocabularies.Source: §1 · p. 2
  • Hierarchical diffusion gives coarse tokens richer semantics than mask or random noise and permits some self-refinement, unlike masked diffusion.Its semantic consistency also makes progressive denoising easier than with uniform random noise.Source: §1 · p. 2
  • HDLM uses a continuous-time Markov chain with transitions restricted to adjacent hierarchies and derives a closed-form diffusion ELBO.The framework supports GIDD- and MDLM-type training and sampling, theoretically and heuristically.Source: §1 · p. 2

2 Preliminary

Discrete diffusion progressively corrupts discrete data through a Markov chain and trains a model to reverse that corruption. The preliminary section describes token-wise categorical transitions and GIDD's flexible interpolation of noise distributions.

2 Preliminary

  • Discrete diffusion corrupts a discrete data point through a forward Markov chain toward a sampleable prior, then learns reverse transitions to recover data.The paper uses a continuous-time Markov chain and sets T = 1.Source: §2 · p. 3
  • D3PM applies independent categorical transitions to each token, with masked diffusion's marginal mixing the original token and a mask token.The cited schedule sets β_t = 1 − α_t.Source: §2 · p. 3
  • GIDD interpolates between data and a smoothly changing distribution π_t, with masked diffusion as the special case π_t = m.Its mixing distribution can combine masking and uniform noise.Source: §2 · p. 3
  • GIDD's masked and uniformly perturbed tokens still lack rich semantics, and its self-correction from uniform noise harms performance versus masked diffusion.The passage attributes correction ability to uniform noise and reports that it reduces performance relative to masked diffusion.Source: §2 · p. 3

3 Hierarchical Diffusion Language Model

HDLM inserts semantically coarse cluster tokens between words and masks, then progressively decodes toward detailed word tokens. The section derives its CTMC and training objective, establishes compatibility with MDLM, and analyzes practical training and sampling choices.

3.1 Forward Process

  • HDLM maps detailed word tokens to higher-level cluster tokens and then masks, with the hierarchy optionally hard-coded or learned.The mappings are predefined by clustering pretrained word-token embeddings in the implementation.Source: §3.1 · p. 3
  • Corrupting tokens into inaccurate clusters with probability controlled by ξ is intended to reduce the training–sampling context gap and support self-correction.The standard process is recovered when ξ = 1; the paper reports 60% smaller generative perplexity than baselines for HDLM with ξ < 1.Source: §3.1 · pp. 4–5
  • The framework supports arbitrary hierarchy depth, but experiments with more than one intermediate hierarchy are left for future work.Multiple levels may allow more generation control but may require more careful sampling design.Source: §3.1 · p. 5

3.2 Reverse Process

  • The reverse process uses the model’s predicted word-level distribution for a noisy input to define reverse transitions.The model predicts over the word-level vocabulary at each timestep.Source: §3.2 · p. 5

3.3 Training

  • HDLM has a closed-form CT-ELBO whose two cross-entropy terms train cluster-level and token-level predictions; expected token- and cluster-loss weights are schedule-invariant.The cluster-level term requires identifying the correct cluster, while the token-level term predicts within that cluster; both expected weights equal 1.Source: §3.3 · p. 6
  • MDLM is the one-cluster special case of HDLM, and its ELBO is recovered within the more general framework.The single cluster has the same embedding as the mask.Source: §3.3 · p. 6

3.4 Example Processes

  • The example processes vary αt = (1 − t)^γ and derive the corresponding cross-entropy weights for γ = 1, 2, and 3.Figures 2 and 3 visualize the marginal probabilities and cross-entropy weights, respectively.Source: §3.4 · p. 7, Figure 2 · p. 7, Figure 3 · p. 7

3.5 Improved Techniques for HDLM Training and Sampling

  • Cluster-level training acts as an easier stage in an easy-to-hard curriculum, while replacing it with whole-vocabulary token loss slightly harms performance.The hard-training variant uses a looser ELBO bound; adding a direct auxiliary cluster-classifier head also mildly reduces performance relative to Bayesian-posterior cluster prediction.Source: §3.5 · p. 8

4 Experiments

HDLM is evaluated on OpenWebText using validation and generative perplexity, with small and base DiT models trained on 131B tokens. Small HDLMs outperform other discrete diffusion variants on both metrics, while ablations examine cluster count and perturbation probability.

4.1 Experimental Setup

  • Experiments train HDLM on OpenWebText (OWT) using DiT small (170M) and base (425M), a GPT-2 tokenizer, 131B training tokens, 512-token contexts, no sentence packing, and validation and generative perplexity metrics.The validation set uses 100,000 of 8,013,769 data points.Source: §4.1 · p. 8

4.2 Main Results

  • Small HDLMs consistently surpass other discrete diffusion counterparts on OWT validation and generative perplexity, while base HDLM matches autoregressive models.Table 1 reports validation perplexity and generative perplexity, with trained-token counts for comparison.Source: Table 1 · p. 9, §4.2 · p. 9

4.3 Ablation Studies

  • Cluster counts around 64 and 128 perform best, while one cluster recovers MDLM performance; validation perplexity first rises and then falls as cluster count increases.The authors suggest a moderate cluster count near the square root of vocabulary size, dividing generation into stages of approximately equal complexity.Source: Table 2 · p. 9, §4.3 · p. 9
  • With stochastic perturbation probability ξ = 0.9 and ξ = 0.8, generative perplexity reduces by over 51% and over 62%, respectively.The authors attribute the benefit to robustness to inaccurate contexts; the reported generative perplexity is 184.0 for ξ = 0.9 without force transition.Source: §4.3 · pp. 9–10, Table 3 · p. 10

5 Related Work

Related work establishes discrete diffusion as an alternative to autoregressive language modeling, while identifying trade-offs in existing noise processes. HDLM instead uses semantic hierarchies to refine fixed-length sequences token by token.

5 Related Work

  • Recent discrete diffusion models including SEDD [20], MDLM [26], and MD4 [30] show effectiveness in small-scale text pretraining, while LLaDA and Dream [38] demonstrate scaling potential on other tasks.Masked diffusion is common for its simplicity and empirical effectiveness, but the paper notes its loss of self-correction ability.Source: §5 · p. 10
  • Uniform-noise models typically underperform masked counterparts, and GIDD [34] combines uniform and masking noise in a general framework.Earlier approaches also use continuous diffusion [18] or combine diffusion with autoregressive models [12] [10].Source: §5 · p. 10
  • HDLM introduces a different noising procedure that the authors describe as conceptually simple and practically effective while preserving self-correction.Unlike block diffusion [2], which generates text blocks autoregressively and diffuses within each, HDLM fixes sequence length and generates token semantics sequentially.Source: §5 · p. 10

6 Conclusion

The conclusion presents HDLM as a flexible discrete diffusion family that progressively decodes abstract token meanings into detailed ones. The paper derives its training ELBO and reports strong language-modeling experiments.

6 Conclusion

  • HDLM is a generic, flexible family of discrete diffusion models that uses hierarchical semantic structures for progressive decoding from abstract to detailed tokens.The paper characterizes this as next semantic scale prediction.Source: §6 · p. 10
  • Using a continuous-time Markov chain framework, the paper develops the forward and backward processes and derives a strict diffusion training ELBO.The authors also formally demonstrate properties of the process.Source: §6 · p. 10
  • The paper proposes design principles and practical training techniques based on theoretical insights and empirical observations, and reports strong language-modeling capability in extensive experiments.Source: §6 · p. 10

A.1 Proof and Analysis of Standard HDLM ELBO

This appendix proves the standard HDLM ELBO and analyzes its special-case relation to MDLM and the behavior of its training-loss weights. The analysis also identifies schedule conditions under which the weights are nonnegative and discusses endpoint limits.

A.1 Proof and Analysis of Standard HDLM ELBO

  • The standard HDLM ELBO is derived for the hierarchical CTMC diffusion process with block conditional transition.The proof evaluates transition cases for mask, cluster, and clean-token states before combining them.Source: §A.1 · pp. 14–15
  • The first ELBO term simplifies to a weighted sum of two cross-entropy losses, and the appendix concludes with the continuous-time ELBO for HDLM.The second term has expectation zero.Source: §A.1 · p. 15
  • MDLM is the special case of HDLM with one cluster equal to the mask, and the appendix shows that its ELBO reduces to the MDLM objective.With all word tokens in the same cluster, the second term is zero and the first term matches MDLM [26].Source: §A.1 · p. 16
  • Under common schedules with decreasing α_t and increasing β_t,m, the loss weights are nonnegative; their expectations are finite, though endpoint limits may not be.For the linear schedule, endpoint weights can be infinite, which the authors consider reasonable when cluster and mask tokens must be fully decoded at t = 0; the analysis supports proper ELBOs under mild conditions.Source: §A.1 · pp. 16–17

A.2 Extended Forward Processes

The extended forward processes generalize HDLM to stochastic cluster perturbations and arbitrary hierarchy levels, with an ELBO that has a weighted cross-entropy interpretation.

A.2 Extended Forward Processes

  • With stochastic perturbations, the ELBO encourages mask-to-cluster and cluster-to-token prediction, plus out-of-cluster exploration when the current cluster is inaccurate.When ξ = 1, this ELBO reduces to the version without perturbations.Source: §A.2 · pp. 17–19
  • For arbitrary hierarchy levels, the forward chain proceeds from clean tokens through intermediate hierarchies to masks, and the reverse process decodes toward more detailed semantics.The arbitrary-hierarchy derivation assumes ξ = 1 for simplicity.Source: §A.2 · p. 19
  • For arbitrary hierarchies, the ELBO decomposes into weighted cross-entropy losses that decode each level into the more detailed level below it.The hierarchy mapping Γ maps tokens at level i to level i + 1.Source: §A.2 · pp. 19–20

B.1 Experimental Settings

Experiments use DiT-based small and base models trained on 512-token contexts; evaluation estimates likelihoods with the ELBO and reports perplexity alongside downstream tasks.

B.1 Experimental Settings

  • The DiT-based SMALL model has 12 layers and 92.1M non-embedding parameters, while BASE has 24 layers and 321.2M.Both use the GPT2 tokenizer.Source: §B.1 · p. 20
  • Models train for 500k steps on 131B tokens with context and batch sizes of 512, using Adam and a cosine-decayed learning rate.The training setup clips loss weights for stability, while ELBO evaluation leaves them unclipped; sequences are windowed or padded to 512 tokens.Source: §B.1 · pp. 20–21
  • Downstream likelihood-based multiple-choice evaluation uses ELBO-estimated per-token likelihoods, while the authors caution that validation and generative perplexity are imperfect measures.Validation perplexity is not an effectiveness metric across different vocabulary sizes, and ELBO is only an upper bound on exact likelihood.Source: §B.1 · p. 21

B.2 Additional Results

Additional results report downstream benchmark performance, schedule and training-mode ablations, clustering behavior, and unconditional generation examples.

B.2 Additional Results

  • Larger γ worsens validation perplexity but improves generative perplexity on OWT, consistent with decoding more tokens later when context is richer.The schedule ablation compares HDLM-64 models trained for 131B tokens.Source: Table 4 · p. 21, §B.2 · p. 21
  • The harder training task slightly hurts performance, and hard training uses a looser ELBO bound than the original HDLM formulation.The evaluation ELBO calculation remains unchanged.Source: §B.2 · p. 21, Table 5 · p. 21
  • On LM1B, HDLM outperforms SEDD and MDLM; its gains on OWT are more significant, suggesting greater advantages on difficult tasks.The LM1B setup trains for 1M steps, totaling 33B tokens.Source: §B.2 · p. 22, Table 6 · p. 22
  • HDLM-small is best on two downstream datasets and its average accuracy exceeds other diffusion models trained twice as long; the large model beats GIDD on every dataset.HDLM-32 with ξ = 0.9 is better than its ξ = 1 counterpart on five of six tasks.Source: Table 7 · p. 22, §B.2 · p. 22

C.1 Clustering algorithm of word embeddings

  • Semantic Clustering uses pretrained token embeddings, K-means++ initialization, cluster-size controls, and converged assignments to define the word-to-cluster mapping.Cluster centroids also initialize the cluster-token embeddings.Source: §C.1 · p. 23

C.2 Examples of Word Clusters

  • Across tested cluster counts, clustering generally converges in one minute, with semantic coherence scores between 0.3 and 0.4 for most settings.The paper shows example words and cluster sizes for 32 clusters.Source: §C.2 · p. 23, Table 8 · p. 26, Table 9 · p. 27

C.3 Discussions

  • Clustering quality considerably affects model performance, and the authors identify more sophisticated or learned hierarchy mappings as future directions.They state that their theoretical framework also applies to hierarchical clustering.Source: §C.3 · p. 23

D Unconditional Generation

  • The paper provides unconditional-generation samples from HDLM-128-base at 512 and 1024 sampling steps.The examples are presented as sample contents rather than as a quantitative evaluation.Source: D Unconditional Generation · pp. 28–29

Related papers

Loading 2510.08632v1…