PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling

Mathias Ollu, Nikos Komodakis

arXiv:2610.08738v1cs.CLcs.LG

TL;DR

Diffusion language models offer parallel text generation, but continuous models still face design challenges in representing and corrupting discrete tokens. H-CDLM jointly denoises token and cluster representations, and H-CoBit improves generation and reasoning results while the framework also improves FLM.

  • Problem

    Continuous diffusion language models have a larger representation-and-corruption design space and have not fully closed the performance gap with autoregressive models.

    Source: §1 · p. 2

  • Method

    H-CDLM jointly diffuses token sequences and cluster IDs from pretrained embeddings with one model, while allowing modality-specific samplers and schedules.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    H-CoBit improves generation quality across benchmarks and reaches 27.4% accuracy on GSM8K, while H-FLM consistently improves over FLM.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The two-level token-and-cluster approach is presented as a first step toward language models that denoise text at multiple levels of abstraction.

    Source: §6 · p. 11

    The authors note that more realistic generated content may make disinformation more convincing.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities. Applied to CoBit, this yields H-CoBit, which delivers large empirical gains across benchmarks. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 24.2 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior continuous diffusion and flow-based models. We further apply H-CDLM to the flow matching model FLM, obtaining consistent gains with H-FLM and demonstrating that the framework generalizes across continuous generative paradigms. Our code will be made publicly available at https://github.com/matol-16/HCDLM.git .

1 INTRODUCTION

Autoregressive language models generate sequentially, while diffusion language models offer parallel denoising but have not fully closed the performance gap with autoregressive models. H-CDLM jointly denoises token and cluster representations, and H-CoBit shows gains across language-generation and reasoning benchmarks.

1 INTRODUCTION

  • H-CDLM jointly diffuses tokens and cluster IDs derived from pretrained token embeddings, with independently tunable samplers and schedules for the two modalities.The framework is also applied to the continuous flow matching model FLM, yielding H-FLM with consistent improvements.Source: §1 · p. 2
  • Clusters provide a coarse semantic anchor that narrows plausible token choices under noise, while token-level predictions refine cluster estimates.Because both modalities are denoised in parallel, information flows in both directions across semantic granularities.Source: §1 · p. 2
  • At data-matched entropy, H-CoBit reduces GenPPL from 73.6 to 49.4 on LM1B and from 71.1 to 50.4 on OWT, while improving MAUVE and surpassing discrete DLMs.The reported GenPPL improvements over CoBit are 24.2 points on LM1B and 20.7 points on OWT.Source: §1 · p. 2
  • H-CoBit reaches 27.4% accuracy on GSM8K, outperforming prior continuous diffusion and flow language models; a random-cluster control does not recover the gains.The control supports the authors’ interpretation that meaningful cluster semantics, rather than added capacity or auxiliary structure, account for the gains.Source: §1 · p. 3
  • The two-level token-and-cluster setup is presented as a first step toward denoising text at multiple abstraction levels, potentially including whole sentences.The introduction also notes that H-CDLM adds little compute or parameter overhead when the number of clusters is small relative to the vocabulary.Source: §1 · pp. 2–3

2 RELATED WORK

The related work situates H-CDLM among discrete hierarchical diffusion, continuous diffusion and flow models, and multimodal joint diffusion. It distinguishes joint hierarchical denoising from approaches that predict or unmask representations in stages.

Discrete Diffusion Models

  • Discrete methods range from masked or uniform token corruption to hierarchical vocabularies and tree-based semantic prediction; H-CDLM is a continuous analog that diffuses modalities jointly.Unlike HDLM and TDLM, it does not unmask clusters before tokens.Source: §2 · p. 3

Continuous Diffusion and Flow Language Models

  • Continuous diffusion and flow models vary in token representations and objectives; CoBit uses bit representations to reduce output-head size and has strong reported results among continuous DLMs.Other approaches include one-hot or trained embeddings, and some avoid predicting actual tokens to keep the output head smaller.Source: §2 · p. 3

Multimodal Joint Diffusion

  • Joint image diffusion combines multiple representations with one denoiser and can improve generation quality; the related work notes that this approach had not been extended to continuous DLMs.Prior image work includes jointly diffusing pixels and DINOv2 features, while a cited image-text setup does not evaluate unconditional text generation.Source: §2 · p. 3

3 BACKGROUND

The background gives a common account of continuous diffusion and flow language models, from embedding discrete tokens in continuous spaces to training denoisers and sampling by reverse integration. It also introduces information-uniform schedules and stochastic churn as sampling adjustments for text.

3 BACKGROUND

  • The background frames continuous diffusion and flow models as a general setting, while noting that S-FLM uses geodesic interpolation on the hypersphere.The stated setting covers models with differing token representations, corruption parameterizations, training objectives, and samplers.Source: §3 · p. 4

3.1 CONTINUOUS DIFFUSION OF DISCRETE TOKENS

  • Continuous diffusion represents tokens in a continuous space with encoder and decoder functions, then adds Gaussian noise to the encoded sequence.The decoder is defined to recover the original sequence from its encoding; the denoiser estimates clean embeddings from noisy ones.Source: §3.1 · p. 4

3.2 TRAINING

  • Training fits a denoiser to estimate clean embeddings from noisy inputs; the training noise density and loss weighting control effort across noise levels.The background describes the objective as regression under a conditional-expectation formulation.Source: §3.2 · p. 4

3.3 SAMPLING

  • Sampling integrates the probability-flow ODE backward over a descending noise grid, decodes the final state, and adjusts text sampling with information-uniform schedules and stochastic churn.For two modalities, Figure 2 shows separate corruption noise levels; churn reinjects noise before a solver step, with its rate and variance set by sampler parameters.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

4 METHOD

The method extends continuous DLMs with jointly diffused token and cluster modalities, then adapts noise schedules and samplers for each modality.

4.1 HIERARCHICAL CONTINUOUS DIFFUSION

  • The joint process concatenates modality embeddings along channels, noises each modality at its own level, and denoises them jointly without increasing sequence length.At sampling time, auxiliary modalities are dropped at the last step when they are deterministic functions of the tokens.Source: §4.1 · pp. 5–6

4.3 PER-MODALITY SCHEDULING AND SAMPLING

  • Figure 3 compares token and cluster schedules, with the median shifting from σ50 = 0.89 to 1.13; cluster identity survives larger noise.The per-modality information-uniform schedule aims for approximately equal conditional information revealed at each step.Source: Figure 3 · p. 6, §4.3 · p. 6

4.2 CONSTRUCTING HIERARCHICAL MODALITIES FOR TEXT

  • Token clusters are assigned position-wise from pretrained token embeddings and continuously encoded; clustering at multiple granularities can produce deeper hierarchies.CoBit uses bit sequences for cluster encodings, while FLM uses one-hot vectors.Source: §4.2 · p. 6

4.3 PER-MODALITY SCHEDULING AND SAMPLING

  • Stochastic sampling can be adapted per modality, and the combined mechanisms yield clear performance improvements; further sampler tuning is left to future work.Sampler mechanisms and parameters can be tuned per modality to improve interplay between modalities.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

H-CoBit improves CoBit’s generation quality, diversity, and downstream accuracy, while ablations examine cluster design and per-modality sampling. H-FLM results show gains also transfer to a continuous flow-matching model.

5.1 EXPERIMENTAL SETUP

  • Experiments train on LM1B, OWT, and TinyGSM; evaluation uses GenPPL–entropy frontiers, MAUVE, and GSM8K accuracy across sampling and NFE sweeps.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • At dataset entropy, H-CoBit reaches GenPPL 57.3 on LM1B, over 20 points better than CoBit-S at 60% of its training steps, and 67.8 on OWT, 15.4 points better.Source: §5.2 · p. 7, Table 1 · p. 9
  • With per-modality churn and snoise = 1.05, H-CoBit reaches GenPPL 50.4 on OWT versus CoBit-S at 71.1; MAUVE is 0.903 vs 0.861 on LM1B and 0.580 vs 0.530 on OWT.Source: §5.2 · p. 7, Table 1 · p. 9
  • On GSM8K, H-CoBit beats other continuous DLMs and narrows the gap to discrete DLMs, outperforming CoBit by 2.1 points at half the NFE and 4 points at matched NFE.Source: §5.2 · p. 7, Table 2 · p. 9

5.3 ABLATIONS

  • At 100k LM1B steps, semantic clusters shift the frontier right while random clusters yield no baseline improvement; n = 16 is best at every entropy level and beats the baseline across loss weights.The retraining ablations are restricted to LM1B at 100k steps because of limited compute; the optimal cluster loss weight is dataset- and cluster-count-dependent.Source: §5.3 · pp. 7–9, Figure 5 · p. 8
  • Late-half cluster churn improves GenPPL by 7.9 and 17.4 points over shared churn on LM1B and OWT, while per-modality scheduling has no clear overall advantage over a shared schedule.The per-modality schedule is slightly better in MAUVE and GenPPL at dataset entropy, but worse at lower entropies.Source: §5.3 · p. 10, Table 3 · p. 10

5.4 CROSS PARADIGM TRANSFERABILITY: H-FLM

  • On LM1B at 1M steps, H-FLM shifts the GenPPL–entropy frontier right and improves GenPPL by 11 points at dataset entropy for NFE=512; stochastic sampling improves both H-FLM and FLM.H-FLM’s winning sampler uses late-half cluster churn, similar to H-CoBit.Source: §5.4 · p. 10, Figure 6 · p. 10

6 CONCLUSION

The paper proposes a multimodal joint diffusion framework for text that adds hierarchical token representations with minimal parameter overhead. Its CoBit and FLM applications show consistent gains, motivating generation across multiple representations from subword units to sentence-level semantics.

6 CONCLUSION

  • The paper proposes a multimodal joint diffusion framework for text with minimal parameter overhead.It integrates hierarchical token representations into continuous diffusion processes, drawing on discrete hierarchical diffusion and multimodal joint image diffusion.Source: §6 · p. 11
  • Applied to CoBit, the multimodal setup yields consistent gains in generation quality and downstream task accuracy, beating recent DLMs in generative quality and continuous DLMs on mathematical reasoning.Source: §6 · p. 11
  • Similar performance gains with FLM support the framework’s cross-model applicability and point toward generating text across representations from subword units to sentence-level semantics.The paper describes its two-level instantiation as a first step toward this broader direction.Source: §6 · p. 11

AI USE STATEMENT

The authors report using generative AI for software code, figures and tables, language review, and text polishing. They state that they reviewed the AI-assisted work and take responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools were used to create or edit software code and to edit figures and tables.Source: AI USE STATEMENT · p. 11
  • The tools were also used to review grammar and formulation and to polish text.Source: AI USE STATEMENT · p. 11
  • The authors state that AI was not used for research directions, proof formulation, hypothesis proposal, or result interpretation, and that they reviewed the assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

The ethics statement notes that stronger generative performance may make disinformation more convincing. The appendix describes the frozen token-to-cluster map and implementation choices used to construct and rebalance it.

ETHICS STATEMENT

  • Generative models can be misused to spread disinformation, and improved generation realism may make it more convincing.Source: ETHICS STATEMENT · p. 11

A SEMANTIC CLUSTERING OF TOKEN EMBEDDINGS

  • The cluster map Γ is a spherical K-means partition of pretrained token embeddings, computed offline, frozen before training, and followed by frequency-driven relabelling.Source: A SEMANTIC CLUSTERING OF TOKEN… · p. 14

A.1 REQUIREMENTS ON Γ

  • Γ is frozen because the cluster sequence must be a deterministic function of the tokens; balanced clusters offer a more expressive anchor, while semantic alignment narrows the token posterior.A uniformly random balanced map does not recover the gain; the appendix attributes the useful signal to tokens sharing clusters being close in embedding space.Source: §A.1 · p. 14

A.2 INPUT EMBEDDINGS

  • For LM1B and OpenWebText, clustering uses normalized LangFlow embeddings; TinyGSM instead uses embeddings from an S-FLM checkpoint.For OpenWebText code tokens, vectors are means of the embeddings for their GPT-2 token spans; tokens without vectors are assigned a separate cluster.Source: §A.2 · p. 14

A.3 CLUSTERING

  • The clustering procedure runs Lloyd’s algorithm with K-means++ initialization, keeps the best of 10 restarts, and stops each run by 300 iterations or an early-shift threshold.The relative centroid-shift threshold is 10^-4.Source: §A.3 · p. 14

A.4 FREQUENCY-DRIVEN CODE REBALANCING FOR H-COBIT

  • Because arbitrary K-means labels can yield imbalanced CoBit bit marginals, the authors search for a frequency-weighted permutation of cluster codes.On OWT, rebalancing raises realized per-bit entropy from 3.75 to 3.99 of 4.0; rebalancing is applied only on OWT and TinyGSM.Source: §A.4 · pp. 14–15

A.5 HYPERPARAMETERS AND MEASURED QUALITY

The appendix reports that the embedding clusters are interpretable but not strongly separated by silhouette scores, and describes a random-cluster control and background connections to diffusion and score matching.

A.5 HYPERPARAMETERS AND MEASURED QUALITY

  • Silhouette values are 0.003–0.075 across maps, so the authors compare them across embedding sources at fixed cluster count rather than treating them as evidence of well-separated clusters.For LM1B with 16 clusters, the partitions separate punctuation, subword continuations, function words, and concrete nouns; the largest cluster consists of unused BERT token IDs.Source: §A.5 · p. 15

A.6 THE RANDOM-CLUSTER CONTROL

  • The random-cluster ablation replaces Γ with an embedding-independent uniform map that is frozen like Γ, isolating the loss of alignment between cluster identity and embedding geometry.Source: §A.6 · p. 15
  • Table 6 describes the seven most-used semantic clusters in OpenWebText at 16 clusters, including their sample shares and frequent code tokens.The listed cluster IDs precede code rebalancing, which permutes labels without changing cluster membership.Source: Table 6 · p. 16

B EXTENDED BACKGROUND

  • The extended background section connects the paper’s general continuous diffusion and flow setup to existing model instantiations that vary in token representation, corruption, objective, and sampler.It identifies S-FLM’s geodesic interpolation on the hypersphere as an exception.Source: B EXTENDED BACKGROUND · p. 16

B.1 AUTO-REGRESSIVE vs DIFFUSION LANGUAGE MODELS

  • Autoregressive models factorize generation left-to-right, whereas diffusion models iteratively refine all positions in parallel; diffusion sampling cost depends on sampling steps rather than only sequence length.Source: §B.1 · p. 16

B.2 LINK TO SCORE MATCHING

  • Under the affine Gaussian path, predicting clean data yields a denoiser up to input rescaling, while a monotone time change preserves the ODE trajectory and changes only the training noise-level density.The denoiser estimates the clean data conditional on the noisy input; schedules control the training noise-level density.Source: §B.2 · p. 16

B.4 FROM OUR GENERAL SETUP TO THE CONTINUOUS DLMS LITERATURE

This section situates continuous DLMs through their shared estimation framework, information-based schedules, and sampling practices. It describes how H-CDLM is instantiated and how generation quality is evaluated.

B.4 FROM OUR GENERAL SETUP TO THE CONTINUOUS DLMS LITERATURE

  • Tables 7–9 place continuous DLMs in a common setup while showing differences in representation, prediction target, objective, noise allocation, and sampler.The models covered include CoBit, LangFlow, FLM, ELF, and S-FLM.Source: Table 7 · p. 17, §B.4 · p. 17, Table 8 · p. 18

B.5 INFORMATION-UNIFORM NOISE SCHEDULES

  • Information-uniform schedules allocate solver steps according to estimated conditional entropy, placing steps uniformly in the schedule’s cumulative distribution.The entropy-rate estimate is approximated using validation loss after a loglinear warmup.Source: §B.5 · pp. 18–19

B.6 SELF CONDITIONING

  • Self-conditioning feeds a previous clean-data estimate back into later denoising steps; prior work argues this can prevent drift from discrete data modes.At sampling, feedback begins after one initial step and adds no sampling steps.Source: §B.6 · p. 19

B.7 INSTANTIATING THE HIERARCHICAL FRAMEWORK

  • H-CDLM can be applied to other continuous diffusion or flow formulations, but empirical evaluation of those variants is left to future work.For k = 2, instantiation requires choosing a cluster encoder, loss weight, and per-modality schedule and sampler.Source: §B.7 · p. 19

B.8 EVALUATING DIFFUSION LANGUAGE MODELS.

  • GenPPL is commonly reported at dataset entropy, but can decrease with sample entropy; MAUVE and downstream task performance provide complementary evaluation measures.The section notes that the pertinence of GenPPL remains debatable even at dataset entropy.Source: §B.8 · p. 19

C OUR METHOD’S DETAILS

The method jointly models the data and its hierarchical modalities, with per-modality information-based schedules and training noise choices. It also identifies broader extensions and links to discrete diffusion.

C.1 THE JOINT DISTRIBUTION IS A VALID MODEL OF THE DATA

  • Jointly modeling tokens with deterministic auxiliary modalities preserves the token data distribution, regardless of coordinate-resolution order or per-modality noise schedules.A sampler of the joint distribution yields a valid token sampler by returning its first coordinate.Source: §C.1 · p. 20

C.2 PER-MODALITY INFORMATION UNIFORM SCHEDULERS

  • Per-modality information-uniform schedules equalize conditional information per step; shared grids implicitly resolve clusters earlier, while coupled grids synchronize information quantiles.The cluster schedule is right-shifted because cluster identity survives larger noise; the coupled per-modality schedule is the default.Source: §C.2 · p. 21

C.3 TRAINING-TIME NOISE SAMPLING.

  • Training noise levels can be sampled independently to cover relative-noise configurations or coupled by a shared quantile to focus capacity on the inference trajectory.The coupled variant is used by default.Source: §C.3 · p. 21

C.4 VARIATIONS AND LINKS

  • The framework supports deeper cluster trees and sequence-level variables, but this work evaluates only the per-token k = 2 construction.Sequence-level variables require their own length and alignment.Source: §C.4 · p. 21
  • Per-modality sampler and parameter tuning can change modality interplay; the method is a continuous-diffusion analogue of HDLM, though all modalities remain jointly diffused.The paper reports substantial sample-quality improvements from varying per-modality samplers for H-CoBit.Source: §C.4 · p. 21

D EXPERIMENTAL DETAILS

The experiments largely retain the base models’ recipes while adding a cluster modality, then evaluate generation quality, efficiency, and a flow-matching instantiation.

D EXPERIMENTAL DETAILS

  • H-CoBit retains CoBit’s recipe and hyperparameters except for the cluster bits and λ2, appending four cluster bits per token in a 12-layer Diffusion Transformer.The denoiser is conditioned on σ1, keeping backbone conditioning identical to the baseline.Source: D EXPERIMENTAL DETAILS · p. 22
  • Training uses AdamW and self-conditioning, fits a separate information-uniform schedule per modality, and sweeps sampling churn jointly with NFE.LM1B and OWT use cosine learning-rate decay; cluster churn is restricted to the late noise window.Source: D EXPERIMENTAL DETAILS · p. 22
  • Evaluation uses GPT-2 Large GenPPL and MAUVE on unconditional samples and exact-match GSM8K accuracy; H-FLM follows FLM’s recipe with added cluster loss and stochastic sampling.For GSM8K, prompts remain noise-free and loss is applied only to answers.Source: D EXPERIMENTAL DETAILS · pp. 22–23

E.1 MAUVE EVALUATION OF H-COBIT

The experiments compare H-CoBit’s quality, efficiency, cluster representation, and per-modality sampling, then describe example generations and the scope of the study.

E.1 MAUVE EVALUATION OF H-COBIT

  • At dataset entropy and matched or lower NFE, H-CoBit improves MAUVE over CoBit by approximately 0.03–0.04 on LM1B and 0.05–0.11 on OWT.Below dataset entropy, CoBit surpasses H-CoBit.Source: §E.1 · p. 23, Table 11 · p. 23

E.2 TRAINING AND SAMPLING SPEED COMPARISON OF H-COBIT

  • H-CoBit has small training and sampling overhead from its larger channel width, but matching CoBit-S quality at lower NFE more than compensates for the speed gap.Table 12 reports the training and sampling timings.Source: §E.2 · p. 23, Table 12 · p. 23

E.3 A CO-BIT SPECIFICITY: APPENDED vs EMBEDDED CLUSTER BITS

  • Embedding cluster indices in token bit codes performs worse than baseline for n = 16, indicating that this approach does not replace an independent cluster representation.The comparison uses LM1B at 100k training steps.Source: §E.3 · p. 23, Figure 7 · p. 24

E.4 TOKEN-CLUSTER AGREEMENT FOR H-COBIT

  • Except at very early sampling steps, the cluster head’s prediction agrees with the final sample more than the token-implied cluster, indicating additional nonredundant information.The token-implied cluster is obtained by marginalizing token probabilities.Source: §E.4 · p. 24

E.5 PER-MODALITY SAMPLING EVALUATION OF H-COBIT

  • Per-modality sampling adjustments produce substantial GenPPL gains at dataset entropy; churn magnitude alone has no effect without late churning.The evaluation varies global noise, churn window, and per-modality churn level.Source: §E.5 · p. 24, Table 13 · p. 25

F GENERATION EXAMPLES

  • The generation examples show unconditional samples from the best-performing LM1B and OWT settings, including samples selected near each batch’s GenPPL.The LM1B examples are 128 model tokens; the OWT examples show the first 1200 characters of 1024-token samples.Source: F GENERATION EXAMPLES · p. 24, Figure 9 · p. 26, Figure 10 · p. 27

G.1 LIMITATIONS

  • The study evaluates one auxiliary modality from fixed pretrained-embedding clusters and relatively small models; whether gains persist at larger scales remains unknown.Richer hierarchies and jointly learned cluster assignments are left unexplored.Source: §G.1 · p. 26

G.2 EXTENSIONS

  • Classifier-free guidance on the cluster modality is proposed as a way to strengthen semantic coherence, but is left for future work.The proposal draws on multimodal image diffusion.Source: §G.2 · p. 26
Loading 2610.08738v1…