PaperlayerView exact revision on arXiv

Source-linked AI summary

CANDI: Hybrid Discrete-Continuous Diffusion Models

Patrick Pynadath, Jiaxin Shi, Ruqi Zhang

arXiv:2510.22510v3cs.LGstat.ML

TL;DR

Continuous diffusion has underperformed discrete formulations on discrete data, despite offering joint updates. CANDI analyzes the mismatch between identity corruption and rank degradation, then decouples discrete masking from Gaussian noise. It outperforms masked diffusion in classifier-guided controlled generation and low-NFE text generation.

  • Problem

    Continuous diffusion has underperformed discrete diffusion on discrete data despite its potential for joint updates, leaving the source of that gap unresolved.

    Source: §1 · p. 1

  • Method

    CANDI analyzes token identifiability and decouples discrete masking from Gaussian corruption to learn conditional structure and continuous scores together.

    Source: §1 · pp. 1–2, §5.1 · p. 6

  • Results

    CANDI outperforms masked diffusion in classifier-guided controlled generation and low-NFE text generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Decoupling corruption lets continuous geometry support discrete generation while retaining conditional dependencies.

    Source: §1 · p. 2, §7 · p. 9

    The empirical validation is framed by a hypothesis that standard continuous diffusion fails at large vocabulary sizes while CANDI remains competitive across vocabulary sizes.

    Source: §6.2 · p. 7

Abstract

from arXiv · show

While continuous diffusion has shown remarkable success in continuous domains such as image generation, its direct application to discrete data has underperformed pure discrete formulations. To understand this gap, we introduce token identifiability, an analytical framework characterizing how Gaussian noise corrupts discrete data through two mechanisms: discrete identity corruption and continuous rank degradation. We reveal that these mechanisms scale differently with vocabulary size, creating a temporal dissonance that forces a tradeoff between learning continuous geometry and discrete structure. To address this, we propose CANDI (Continuous ANd DIscrete diffusion), a hybrid framework that decouples discrete and continuous corruption, enabling simultaneous learning of both. This unlocks the benefits of continuous diffusion for discrete spaces: on controlled generation, CANDI enables classifier-based guidance with off-the-shelf classifiers through simple gradient addition; on text generation, CANDI outperforms masked diffusion at low NFE, demonstrating the value of learning continuous gradients for discrete spaces. We include the code on the project page: https://patrickpynadath1.github.io/candi-lander.

1. Introduction

Continuous diffusion offers joint position updates but has underperformed discrete diffusion on discrete data, despite its potential advantages at low NFE. CANDI separates discrete and continuous corruption to learn both conditional structure and continuous geometry, outperforming masked diffusion in the reported settings.

1. Introduction

  • Discrete diffusion has generally outperformed direct continuous-diffusion applications to discrete spaces, despite continuous diffusion's joint updates and potential value at low NFE.Discrete frameworks have also scaled to large language models and performed strongly across text-generation tasks.Source: §1 · p. 1
  • Token identifiability analyzes Gaussian corruption through whether the correct token remains the argmax and how much its signal degrades.These two axes connect discrete identity corruption to learning conditional dependencies and rank degradation to learning a continuous score function.Source: §1 · p. 1
  • As vocabulary size grows, discrete identity corruption and continuous rank degradation diverge, creating noise-level dissonance that prevents learning continuous scores and discrete dependencies together.At noise levels preserving identifiable tokens, continuous denoising is trivial; at levels that degrade the continuous signal, too few tokens remain identifiable.Source: §1 · pp. 1–2
  • CANDI combines discrete conditional structure with joint updates, capabilities that the figure contrasts with discrete-only and continuous-only diffusion.The proposed hybrid is presented as retaining both capabilities.Source: Figure 1 · p. 2
  • CANDI outperforms masked diffusion in classifier-guided controlled generation and low-NFE text generation, using continuous geometry for gradient addition and joint position evolution.The framework combines Gaussian noise on some positions with discrete masking on others.Source: §1 · p. 2

2. Related Works

Prior work includes discrete corruption processes and methods that apply continuous diffusion to discrete sequences. These approaches either lack continuous geometry or address different barriers than CANDI.

2. Related Works

  • Masked and uniform discrete diffusion use categorical corruption but lack the continuous geometry associated with Gaussian diffusion.Masked methods replace tokens with a mask; uniform methods sample from the vocabulary.Source: §2 · p. 2
  • Embedding-based continuous methods inject Gaussian noise into learned token embeddings, while one-hot Gaussian diffusion by Han et al. (2023) uses semi-autoregressive generation.Embedding-based approaches must address degenerate embedding issues; CANDI is fully non-autoregressive and focuses on the mismatch between discrete and continuous corruption.Source: §2 · p. 2
  • Sahoo et al. (2025) connect uniform state and continuous Gaussian diffusion to motivate curriculum learning and distillation, while CANDI examines barriers to Gaussian diffusion on discrete spaces.The paper also notes independent hybrid discrete-continuous diffusion efforts in Appendix A.Source: §2 · p. 2

3. Preliminaries

The preliminaries define categorical sequences and review masked discrete and Gaussian continuous diffusion. Both processes use conditional expectations to support reverse generation, while continuous diffusion also admits deterministic probability-flow sampling.

3. Preliminaries

  • A length-l sequence over |V| categories is represented by categorical tokens X and a sequence of one-hot vectors X̃.The notation distinguishes discrete tokens from their vector representations.Source: §3 · p. 2
  • Masked diffusion gradually replaces ground-truth tokens with a mask, and learning E[X0|Xt] is sufficient to simulate its reverse process.The forward process interpolates between the data distribution and the all-mask distribution.Source: §3 · pp. 2–3
  • Continuous diffusion uses a variance-exploding SDE with diffusion coefficient g(t), and deterministic sampling uses a probability-flow ODE without the stochastic term.The forward SDE adds Gaussian noise to the one-hot sequence.Source: §3 · p. 3
  • For continuous diffusion, the score-function parameterization uses the conditional expectation E[X̃0|X̃t] and the cumulative noise σ(t).The supplied passages state that learning this expectation is sufficient to simulate the reverse process.Source: §3 · p. 3

4. A New Lens: Token Identifiability under Gaussian Noise

Token identifiability frames Gaussian corruption as two distinct signals: whether the correct token remains identifiable and how its relative rank degrades. Their differing dependence on vocabulary size creates a temporal mismatch that obstructs learning both conditional structure and continuous scores.

4. A New Lens: Token Identifiability under Gaussian Noise

  • The analysis characterizes both corruption axes and identifies temporal dissonance as vocabulary size increases, hindering continuous diffusion on categorical data.The paper includes a visual summary in Figure 2.Source: §4 · p. 3

4.1. Definitions

  • Token identifiability describes how the noisy latent relates to the correct token through two complementary measures: identity corruption and continuous rank degradation.The former asks whether discretization yields the correct token; the latter measures the fraction of incorrect tokens ranked above it.Source: §4.1 · p. 4

4.2. Why Continuous Diffusion Fails on Discrete Data

  • Discrete identity corruption governs learning token dependencies because clean anchor tokens are needed to learn co-occurrence patterns in sequences without geometric continuity.The passage contrasts conditionally related words such as “New” and “York,” which have different token identities.Source: §4.2 · p. 4
  • Continuous rank degradation supplies the denoising signal for a continuous score, enabling coordinated refinement across positions, especially when few function evaluations require parallel updates.Without that score, updates rely on independent conditional sampling, which can produce incoherent generations at low NFE.Source: §4.2 · p. 4
  • Effective discrete generation requires both conditional structure and continuous gradients; Gaussian noising alone creates a vocabulary-dependent temporal dissonance between them.The paper relates this mismatch to prior augmentations including self-conditioning, prefix completion, and random infilling, while directing discussion of those techniques to an appendix.Source: §4.2 · pp. 4–5
  • Discrete identity corruption grows problematically with vocabulary size while continuous rank degradation remains category-independent, so their useful noise levels become misaligned.Under standard Gaussian noising at large vocabulary sizes, score-learning noise levels do not preserve enough identifiable tokens for learning conditional structure.Source: §4.2 · p. 5

5. CANDI: Continuous And Discrete diffusion

CANDI addresses temporal dissonance by separating discrete masking from Gaussian corruption so models can learn both discrete conditional structure and continuous scores. Its design includes a structured noising kernel, masked-diffusion parameterization, and hybrid inference.

5. CANDI: Continuous And Discrete diffusion

  • CANDI decouples discrete corruption from Gaussian noise, removing vocabulary-size-dependent scaling and making both corruption processes vary linearly with time.The design aims to align discrete identity corruption with continuous rank degradation across time.Source: §5 · p. 5, §5.1 · p. 6

5.1. Structured Noising Kernel

  • The structured kernel preserves selected clean anchor tokens while applying Gaussian noise to other positions, allowing the model to learn conditional structure and continuous denoising together.A linear keep-rate schedule sets discrete corruption to ρ(t) = t; the continuous schedule targets linearly increasing rank degradation, with the optimal relation between schedules left for future work.Source: §5.1 · pp. 5–6

5.2. Model Parameterization and Training

  • The model learns Pθ(X0|Xt) with the masked-diffusion objective, which supports simulation of both discrete transitions and the continuous score-based reverse trajectory.The method combines masked-diffusion updates with reverse probability-flow ODE updates for positions that remain corrupted.Source: §5.2 · p. 6, §5.3 · p. 6

5.3. Hybrid Inference

  • Hybrid inference samples tokens from the learned conditional distribution to approximate the noisy one-hot expectation with embedding lookup, avoiding expensive full one-hot matrix multiplications.This approximate procedure is intended to retain continuous-score benefits while matching the efficiency of pure discrete methods.Source: §5.3 · pp. 6–7
  • Classifier guidance adds the desired-class input gradient to the learned score function, enabling controllable generation without diffusion-specific classifiers.The inference method also combines masked-diffusion ancestral updates with deterministic ODE updates.Source: §5.3 · pp. 6–7

6. Experimental Results

The experiments assess generative quality with entropy-perplexity frontiers and test CANDI across vocabulary sizes, low-NFE text generation, and classifier-guided molecular generation. They report that CANDI performs well against discrete diffusion baselines across these settings.

6. Experimental Results

  • The experiments use Text8 and OpenWebText as examples of small- and large-vocabulary data to test temporal-dissonance predictions, and report improved low-NFE generation and gradient-based control.The reported comparisons include frontiers for both datasets and classifier-guided QM9 generation.Source: §6 · p. 7, §6.2 · pp. 7–8

6.1. Why Frontier Analysis is Necessary

  • Entropy-perplexity frontiers compare diversity and fluency across sampling temperatures, since small temperature changes can reverse methods’ relative rankings.The study focuses on finite-timestep generation rather than likelihood metrics that assume infinitesimal steps.Source: §6.1 · p. 7

6.2. Empirical Validation of Temporal Dissonance

  • On Text8, continuous diffusion frontiers are fairly close to MDLM and CANDI, but on OWT one-hot diffusion is orders of magnitude worse and embedding diffusion appears mode-collapsed.CANDI outperforms MDLM on both datasets; Figure 6 reports frontiers across the visualized NFE settings.Source: §6.2 · p. 8, Figure 6 · p. 8

6.3. Improved Low-NFE Generation Quality: OWT

  • CANDI has a strictly better frontier than MDLM up to 64 NFE, matching it at 64, and strictly exceeds DUO as perplexity falls below approximately 40.The OWT entropy-perplexity frontiers are shown at NFE 8, 16, 32, and 64.Source: §6.3 · p. 8, Figure 7 · p. 9

6.4. Controlled Generation: QM9

  • CANDI strictly dominates MDLM and achieves competitive performance against UDLM in QM9 controlled generation without requiring a specialized classifier.The evaluation compares target-attribute coherence against the number of unique, valid molecules.Source: §6.4 · p. 9, Figure 8 · p. 9

7. Conclusion

The paper attributes continuous diffusion’s underperformance on discrete data to temporal dissonance between discrete and continuous corruption. It presents CANDI as a hybrid approach that addresses this mismatch and empirically verifies that continuous diffusion can succeed in discrete spaces.

7. Conclusion

  • Temporal dissonance between discrete and continuous corruption is identified as the root cause of continuous diffusion’s underperformance on discrete data.The paper introduces token identifiability to characterize the effect of continuous noise on discrete data.Source: §7 · p. 9
  • CANDI decouples the corruption mechanisms to enable simultaneous learning of discrete conditional structure and continuous score functions.Source: §7 · p. 9
  • The paper empirically verifies temporal dissonance and shows that CANDI resolves it, allowing continuous diffusion to succeed in discrete spaces.Source: §7 · p. 9

A. Concurrent Works

The authors distinguish their work from concurrent hybrid diffusion proposals by emphasizing its explanation of continuous diffusion’s underperformance, its low-NFE and guidance benefits, and its use of frontier analysis for generative quality.

A. Concurrent Works

  • CADD (Zheng et al., 2025) augments masked diffusion with continuous embedding diffusion, while CCDD (Zhou et al., 2025) combines discrete and continuous diffusion in a joint multi-modal framework.CADD is motivated by masked diffusion’s “information void”; CCDD motivates the combination through continuous diffusion’s greater expressivity and uses a factorized reverse process.Source: A. Concurrent Works · p. 13
  • Unlike Zhou et al. (2025), which discusses an expressivity-trainability tradeoff, this work uses token identifiability and temporal dissonance to explain when continuous diffusion fails on categorical data.Source: A. Concurrent Works · p. 13
  • The authors highlight continuous diffusion’s low-NFE generation quality and simple classifier-based guidance as benefits not discussed in Zheng et al. (2025) or Zhou et al. (2025).Source: A. Concurrent Works · p. 13
  • The work uses frontier analysis as its primary measure of generative quality and presents it as applicable beyond hybrid diffusion.Source: A. Concurrent Works · p. 13

B.1. Alternative Derivation of Discrete Identity Corruption

The appendix analyzes discrete identity corruption and token identifiability across prior methods, embedding diffusion, and latent diffusion, including how temporal dissonance behaves in these settings. It also details CANDI’s reverse transitions, inference procedures, and compatibility with off-the-shelf classifier guidance.

B.1. Alternative Derivation of Discrete Identity Corruption

  • For t > 0, argmax-projected Gaussian corruption becomes state-dependent and diverges from discrete uniform-state diffusion because argmax depends on continuous magnitudes absent from discrete data.Without assumptions on the token indices, overtaking probabilities cannot be simplified; the correct-token distribution also cannot be reduced beyond N(˜Xt[i], σ(t)2).Source: §B.1 · p. 14

B.2. Continuous Rank Degradation

  • Rank degradation measures how many incorrect tokens outrank the correct token, capturing categorical information loss that standard SNR misses.Unlike absolute signal magnitude, relative rank reflects recoverability when embeddings collapse into nearly identical regions, even if their norms are large and noise is small.Source: §B.2 · p. 15
  • Across vocabulary sizes 5, 50, 500, and 50,000, simulations closely track the analytical corruption curves, confirming that identity corruption depends exponentially on vocabulary size while rank degradation is vocabulary-independent.The simulation tests 5,000 points at logarithmically spaced variance levels from 0.1 to 10; Figure 9 compares empirical results with analytical predictions.Source: Figure 9 · p. 15, §B.2 · p. 15

B.3.1. CONTINUOUS DIFFUSION FOR SMALL VOCABULARY

  • For small vocabularies, continuous and discrete corruption behave similarly, consistent with Gaussian diffusion performing well on binary data (Chen et al. (2022)).Self-conditioning improved performance on top of already viable results in that work’s ablations.Source: §B.3.1 · pp. 15–16

B.3. Connection to Previous Continuous Diffusion Approaches

  • For larger categorical spaces, divergence between identity corruption and rank degradation motivates examining why prior continuous-diffusion design choices help.The framework presents self-conditioning and masking as approaches relevant to this mismatch.Source: §B.3.2 · p. 16

B.3.2. CONTINUOUS DIFFUSION FOR CATEGORICAL SPACES

  • The framework interprets self-conditioning as using prior predictions as pseudo-clean anchors, while random masking teaches bidirectional dependencies needed for whole-sequence refinement.Prior work reports that random and prefix masks outperform prefix-only training for prefix completion; the framework treats both masking and self-conditioning as ways to provide identifiable conditioning signals.Source: §B.3.2 · p. 16

B.4. Extension to Embedding Diffusion

  • Embedding diffusion exhibits the same asymmetry: discrete embedding corruption scales exponentially with vocabulary size, while continuous rank degradation depends on vocabulary size only through averaging.The embedding corruption derivation uses a general alignment or distance function; the stated limitation applies whether tokens are obtained by dot-product similarity or L2 distance.Source: §B.4 · pp. 18–19

B.5. Empirical Simulation for Embedding Diffusion

  • Simulations with random i.i.d. and learned GPT-2 embeddings show closely matching corruption curves, supporting comparison independence for learned embeddings.The independence assumption is provable for i.i.d. random embeddings but not evident for structured, correlated learned embeddings.Source: §B.5 · p. 19
  • Learned embeddings show temporal dissonance: discrete corruption becomes more sensitive to small noise as vocabulary size grows, while continuous rank degradation stays roughly constant.GPT-2 embeddings behave like i.i.d. random embeddings across vocabulary sizes, supporting the independence assumption used to analyze temporal dissonance.Source: Figure 10 · p. 20, §B.5 · p. 20

B.6. Relation to Latent Diffusion

  • Latent diffusion avoids temporal dissonance because each decoded position can use the entire latent sequence; its pretrained components learn discrete conditional structure before a separate score model.Latent methods’ encoders and decoders are trained using masked language modeling (Meshchaninov et al., 2025).Source: §B.6 · pp. 20–21

C.1. Derivation of Structured Kernel Reverse Transition

  • The reverse kernel has three nonzero mask transitions: clean positions remain fixed, unmasked positions follow the Gaussian reverse process, and noisy-to-clean transitions marginalize over X0.Source: §C.1 · p. 21

C.2. Full Exact Algorithm

  • Exact joint inference carries over clean positions, applies the score-based ODE update to noisy positions, and uses the model’s predicted X0 distribution for discrete conditioning.The exact procedure requires multiplying the predicted expectation by the embedding table.Source: §C.2 · p. 22

C.3. Details on Sampling Trick

  • A single token sample from Pθ(X0|Xt) approximates the costly embedding-table multiplication, and one Monte Carlo step suffices for good generation quality; this also extends Cetin et al. (2025) from embedding to one-hot diffusion.Source: §C.3 · p. 23

C.4. Details on Classifier-based Guidance

  • CANDI supports classifiers trained without diffusion-specific modifications by adding a scaled classifier gradient to the continuous reverse-step update.Source: §C.4 · pp. 23–24

C.5. Parameterization and Training Details

CANDI trains a posterior predictor with a loss weighted by the number of corrupted positions, and uses a learnable bias to distinguish corrupted from clean positions. Its preconditioning preserves clean-position magnitudes while stabilizing noisy inputs across timesteps.

C.5. Parameterization and Training Details

  • The training objective weights prediction loss by the number of corrupted positions, giving each corrupted position equal weight across time steps.The objective corresponds to the ELBO for standard masked diffusion, with a different forward noising kernel.Source: §C.5 · p. 24
  • A learnable bias vector marks corrupted positions; setting λ = 1 gives all corrupted positions the same bias embedding and recovers masked diffusion behavior.In practice, λ = .5; clean positions are embedded separately.Source: §C.5 · p. 24
  • Preconditioning uses the mask to preserve clean-position magnitudes and rescale noisy positions for stable input magnitudes across timesteps.The design follows Karras et al. (2022).Source: §C.5 · p. 24

D.1. Frontier Analysis

The appendix explains frontier-based evaluation and details vocabulary-scaling experiments. It argues that temperature sweeps are needed for fair generative comparisons, then describes end-to-end embedding diffusion experiments and their evaluation setup.

D.1. Frontier Analysis

  • Likelihood metrics are avoided because they assume infinitesimal steps, while ELBO comparisons either favor CANDI’s discrete component or compare different spaces.The experiments instead focus on practical generation with a finite number of timesteps.Source: §D.1 · pp. 24–25
  • Frontier analysis plots entropy–perplexity operating points across sampling temperatures, capturing diversity–fluency trade-offs and method-specific temperature optima.A single fixed temperature may miss a method’s full operating range; the paper notes that different tasks can also favor different temperatures for one model.Source: §D.1 · p. 25
  • Small temperature changes can reverse method rankings, so the authors define a genuine win as frontier dominance across all entropy levels.Figure 11 varies temperatures within .875–1.0; similar entropy ranges alone do not prevent ranking flips.Source: §D.1 · p. 25, Figure 11 · p. 25

D.2. Vocabulary Scaling Experimental Details

  • Text8 and OWT use a 768-dimensional, 12-block transformer; models receive roughly equal training steps, and generated words count as valid only if present in train or test data.The reported runs use 8 H100s; Text8 training takes about 20 minutes and OWT a little over 12 hours.Source: §D.2 · p. 26
  • Because embedding space lacks a closed-form continuous rank-degradation solution, embedding diffusion uses a VE SDE with σmin = .01 and σmax = 2.0.The schedule is used for both Text8 and OWT.Source: §D.2 · p. 26
  • The vocabulary-scaling experiments exclude pretrained embeddings and learn structure end-to-end, keeping the comparison aligned with discrete baselines trained from scratch.Pretrained embeddings would supply external discrete structure and make success or failure ambiguous for the paper’s question.Source: §D.2 · p. 26

D.3. OWT Text Generation

The appendix reports additional OWT frontier, throughput, and inference comparisons, followed by QM9 guidance settings. CANDI’s relative frontier performance varies with NFE, and the observed lack of frontier expansion at higher NFE may depend on model scale.

D.3. OWT Text Generation

  • OWT Figure 6 uses partially trained checkpoints, whereas Figure 7 reflects fully trained model ability; Figure 7 samples temperatures from .7 to 1.0.The sweep uses 128 samples per method and NFE setting.Source: §D.3 · pp. 26–27
  • Autoregressive models gain a throughput advantage over diffusion once NFE exceeds approximately 200, in the GPT-2 and MDLM comparison.The figure uses GPT-2 and MDLM as representative autoregressive and diffusion models.Source: Figure 12 · p. 27
  • At higher NFE, CANDI slightly underperforms as fewer tokens are updated simultaneously, reducing the benefit of gradient information.The authors report that CANDI remains competitive with strong pure-discrete baselines, though it does not achieve the best frontier.Source: §D.3 · p. 27, Figure 13 · p. 27
  • Past 128 NFE, selected methods show no significant frontier improvement; the authors say temperature tuning can achieve the gains, but larger-model persistence is unknown.The authors caution that the observation may be specific to the tested model scale.Source: Figure 14 · p. 28, §D.3 · p. 28
  • The approximate inference algorithm does not degrade generative quality relative to the exact variant on the entropy–perplexity frontier.The paper notes this aligns with Cetin et al. (2025), who approximate the expectation with single-step Monte Carlo entropy.Source: §D.3 · p. 28

D.4. QM9 Guidance Details

  • For QM9, CANDI’s classifier is trained without input corruption to match an off-the-shelf external classifier, and guidance frontiers are evaluated at NFE 16 and 32.The reported CANDI frontier uses τ = .7 across both properties and NFEs.Source: §D.4 · p. 29, Figure 16 · p. 30

D.5. Generated Samples

The appendix reports CANDI sample perplexity and entropy at six NFE settings, with τ = .85 for NFE 16–64 and τ = .875 for NFE 128–512.

D.5. Generated Samples

Related papers

Loading 2510.22510v3…