PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

Ye Yuan, Weien Li, Rui Song, Zeyu Li, Haochen Liu, Xiangyu Kong, Zixuan Dong, Linfeng Du, Zipeng Sun, Weixu Zhang, Jiaxin Huang, Changjiang Han, Yonghan Yang, Zichen Zhao, Xiuyuan Hu, Haolun Wu, Yankai Chen, Fengran Mo, Jikun Kang, Bowei He, Philip S. Yu, Xue Liu

arXiv:2607.13431v1cs.LGcs.AIcs.CL

TL;DR

Discrete diffusion models lack a unified account of how discrete state-space construction shapes their formulations and trade-offs. This survey organizes the field around that design axis and concludes that co-adapted state-space, corruption, objective, and sampling choices support parallel refinement, infilling, editing, and constraint-aware generation.

  • Problem

    Discrete diffusion research lacks a unified design space connecting state-space construction, domain-specific structures, formulations, objectives, inference, and systems considerations.

  • Method

    The survey organizes discrete diffusion models around state-space construction and a common four-component structure spanning corruption, denoising, training, and sampling.

  • Results

    The survey concludes that co-adapting state-space, corruption, learning objective, and sampling choices best matches data structure and task demands.

  • Takeaways & Limitations

    Discrete diffusion most clearly supports parallel refinement, infilling, editing, and constraint-aware generation across domains.

  • Takeaways & Limitations

    Standardized protocols for reporting iterative self-correction remain underdeveloped.

Abstract

from arXiv · show

Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.

1 Introduction

The introduction contrasts autoregressive generation with discrete diffusion’s parallel, globally revisable refinement, while emphasizing that discrete diffusion requires design choices specific to categorical state spaces. It presents tokenization as the organizing principle for a unified framework spanning formulations, domains, systems, evaluation, and open problems.

  • Motivation: Autoregressive models offer stable maximum-likelihood training and mature decoding, but generate discrete sequences through left-to-right factorization.These properties supported the scaling of language models from billion-parameter systems to frontier models with hundreds of billions of parameters.
  • Motivation: Discrete diffusion instead iteratively denoises all positions simultaneously, using bidirectional context and global revision throughout generation.Parallel updates can make generation time largely independent of sequence length given sufficient compute, while global context supports holistic planning and long-range coordination.
  • Problem: Applying diffusion to discrete sequences is nontrivial because categorical spaces lack the Euclidean perturbations and directly applicable score functions used in continuous diffusion.Discrete sequences occupy a categorical space {1, . . . , K}L rather than the same Rd space used by continuous diffusion processes.
  • Core lens: The paper treats tokenization as a first-class design axis that shapes corruption, denoising difficulty, sequence length, and controllability across discrete diffusion models.The framework analyzes vocabulary design, codebook topology, and natural alphabets, and introduces diffusion-facing diagnostics for tokenization quality.
  • Framework scope: The unified framework decomposes discrete diffusion into corruption operator, denoiser parameterization, training objective, and sampler across multiple domains and formulation families.It also treats scaling, training pipelines, inference acceleration, evaluation protocols, and open challenges as integral parts of the design space.

2 Related Work

Prior surveys cover diffusion generation across text, multimodal, graph, and biomolecular domains, but typically do not treat tokenization and domain-intrinsic discrete structure as central organizing principles. This paper instead centers tokenization and discrete state-space diagnostics, extending the framework into practical cross-domain design guidance.

  • Prior surveys: Existing surveys organize diffusion literature around generation paradigms, continuous-versus-discrete formulations, model families, or domain applications rather than tokenization-centric state-space construction.Tokenization is generally treated as a local modeling choice, future challenge, or subtopic within broader taxonomies.
  • Limitations: Prior work under-emphasizes how tokenization and domain-intrinsic discrete structures shape corruption design, reverse parameterization, controllability, and validity across languages, code, multimodal data, biology, and molecules.The paper identifies this cross-domain treatment as a missing link in existing overviews.
  • This paper’s framework: This paper makes tokenization, state-space factorization, and representation diagnostics primary organizing principles, arguing that consequential discrete-diffusion choices arise before and beyond the denoiser architecture.Its framework is positioned as distinct from approaches centered only on model families.
  • Practical contribution: The paper provides cross-domain checklists for selecting tokenization schemes, corruption operators, training objectives, reverse parameterizations, samplers, constraints, and evaluation protocols.The checklists are intended to make the survey directly reusable as a practical design reference.
  • Scope and selection: The work is a narrative survey assembled through citation tracking around foundational discrete-diffusion formulations and major venues and preprint servers through early 2026.Selection prioritizes distinct corruption, parameterization, objective, or sampler choices; new modality or scale regimes; and reusable evaluation results.

3 Background and Notation

This section positions discrete diffusion as a probabilistic, iterative-refinement alternative to autoregressive generation and introduces the discrete state-space and corruption formalism underlying later models.

  • Autoregressive factorization: Autoregressive models factorize discrete-token sequences into left-to-right conditionals trained with teacher-forced per-token cross-entropy.Each conditional is typically parameterized by a causal Transformer over the finite vocabulary V.
  • Autoregressive limitations: AR decoding requires L serial forward passes and permanently commits emitted tokens, limiting correction of early errors and handling of globally spanning constraints.KV caching and speculative decoding reduce per-step cost but do not eliminate the sequential dependency.
  • Iterative refinement: Non-autoregressive methods generate or refine all positions simultaneously, while diffusion formalizes iterative refinement through learned reverse denoising after progressive corruption.The denoising-step count T trades off sample quality against generation cost, and diffusion replaces hand-designed re-masking heuristics with a probabilistic process.
  • Discrete diffusion motivation: Embedding discrete tokens before Gaussian diffusion creates a geometry mismatch, motivating diffusion defined natively over discrete states with categorical corruption and vocabulary-level predictions.Native discrete corruption can use masking, substitution, or structured transitions.
  • State space and forward process: The paper represents sequences as x ∈ V^L, with vocabulary size K and optional one-hot token vectors, and defines corruption as a T-step categorical Markov chain of transition matrices.The cumulative transition matrix allows direct sampling of x_t from x_0, while the terminal distribution approaches a known prior such as uniform over V or concentrated on m.
  • Reverse parameterization: The dominant x_0-parameterization predicts pθ(x_0 | x_t, t), from which the reverse transition is analytically recovered using the known forward posterior.Alternative parameterizations include predicting x_{t−1}, log-probability ratios, or a score analog.

4 Discrete State Spaces and Tokenization

This section presents tokenization and discrete state-space construction as first-class design axes for discrete diffusion, shaping corruption, denoising, computation, and evaluation. It organizes the design space across semantic tokens, quantized media codes, and natural scientific alphabets.

  • 4.1 Cross-domain definition: A token is a finite-vocabulary categorical symbol, and discrete diffusion corrupts and denoises sequences of such symbols through categorical transitions.The forward process replaces tokens or maps them to an absorbing state, while the reverse process predicts a distribution over the vocabulary at each position.
  • Tokenization families: Semantic tokens lack intrinsic metric structure, whereas quantized media codes inherit codebook geometry and scientific alphabets carry domain-specific similarity and validity structure.These differences motivate masking for text, structured corruption toward nearby media codes, and externally guided transitions for scientific data.
  • Why tokenization matters: Tokenization defines perturbation topology because the transition matrix implicitly determines which discrete corruptions count as small or large.Uniform transitions erase locality, while large transition spaces can cause combinatorial explosion and numerical instability; subword fragmentation can also disrupt semantic consistency.
  • Natural scientific alphabets: Scientific representations trade expressiveness against denoising burden: atom-level molecular graphs preserve chemical detail, while fragment-, scaffold-, or motif-level units reduce effective sequence complexity.Coarser units encode chemically meaningful substructures but change the generation granularity.
  • 4.5 Tokenization diagnostics: Tokenization diagnostics should combine information-theoretic and compression metrics, geometric and topological probes, and diffusion-facing measures of generative interaction.The goal is to assess information preservation, semantic structure, and suitability for iterative denoising beyond downstream task performance.
  • 4.5 Tokenization diagnostics: For media, PSNR and SSIM measure low-level reconstruction fidelity, while LPIPS more faithfully captures semantic preservation; schedule sweeps expose tokenizer–noise-schedule coupling.Robust tokenizations maintain sample quality across linear, cosine, and log-linear schedules, simplifying development.

5 Core Formulations of Discrete Diffusion

Discrete diffusion formulations differ mainly in their corruption operators and denoiser parameterizations, while sharing a common reverse-process structure. Masking, substitution, structured transitions, and score/ratio views define complementary points in this design space.

  • 5.1 Discrete-Time Markov Chains and Transition Matrices: D3PM defines discrete diffusion through transition matrices Qt and cumulative matrices ¯Qt, yielding closed-form reverse posteriors used to construct reverse transitions.The model predicts x0 and plugs that prediction into the analytical posterior rather than learning the reverse distribution directly.
  • 5.1 Discrete-Time Markov Chains and Transition Matrices: Uniform substitution treats all token replacements as equally plausible, making denoising difficult because corrupted tokens lack structural clues among K −1 alternatives.Embedding-aware transitions instead favor semantically nearby tokens, but D3PM found them harder to tune and less stable than absorbing-state corruption.
  • 5.1 Discrete-Time Markov Chains and Transition Matrices: Absorbing-state corruption replaces tokens with a persistent mask token, turning denoising into fill-in-the-blank prediction and scaling most successfully in practice.Its stationary distribution concentrates all mass on the absorbing mask state, while training becomes a reweighted masked-language-modeling loss.
  • 5.3 Substitution Noise and Mixture Corruptions: Hybrid corruptions combine masking and substitution, with small substitution components generally improving generation quality while the optimal mixing ratio depends on domain and scale.Pure masking remains competitive at large scale when paired with modern training recipes.
  • 5.5 Score/Ratio-Based Characterization of Reverse Transitions: Score, ratio, predict-x0, and predict-logits formulations are equivalent parameterizations under sufficient denoiser expressivity and shared factorization assumptions.The choice among them therefore concerns parameterization convenience rather than fundamental model expressivity.
  • 5.6 A Unifying View: All discrete diffusion models share four components: a corruption operator, a denoiser parameterization, a training objective, and a sampler.Transition matrices or rate matrices specify corruption, while masking, substitution, structured transitions, and hybrids instantiate particular operators.
  • 5.6 A Unifying View: Formulation differences lie primarily in the corruption operator and denoiser parameterization, whereas training objectives and samplers are often interchangeable or combinable.For example, score-entropy training can be paired with ancestral sampling or τ-leaping algorithms.
  • 5.6 A Unifying View: Corruption choice is orthogonal to parameterization choice, so their optimal pairing is an empirical question shaped by domain, vocabulary size, and computational budget.The framework also shows that apparently distinct models may differ in only one or two components.

6 Training Objectives and Parameterizations

Training objectives in discrete diffusion range from likelihood-centered ELBO optimization to theoretically grounded and highly simplified denoising losses, with trade-offs between principled likelihood estimation, scalability, and generation quality. Parameterization choices likewise determine whether models remain compatible with forward dynamics, use vocabulary capacity efficiently, and support practical sampling.

  • Likelihood and denoising objectives: Masked-diffusion ELBOs can reduce to reweighted cross-entropy on masked positions and, in continuous time, become weighted masked-language-modeling objectives largely invariant to the noise schedule’s interior form.These derivations show that the generative objective can be trained with standard cross-entropy machinery and depends primarily on schedule endpoints in important masked settings.
  • Likelihood and denoising objectives: ELBO optimization provides a principled route to likelihood estimation, whereas simplified denoising objectives can scale more favorably and improve generation quality despite looser variational bounds.Plaid demonstrates maximum-likelihood scaling with a learned noise schedule, while Uniform State Diffusion Models provide the counterpoint favoring simplified objectives.
  • Likelihood and denoising objectives: Further simplifications restrict supervision to corrupted tokens, use linear reweighting, directly optimize token-level cross-entropy, or rewrite masked-diffusion NELBOs using the masked-token count n with 1/n weighting.These approaches remove unnecessary supervision, avoid continuous-to-discrete rounding mismatch, or weaken the need for explicit scalar time conditioning.
  • Conditioning and parameterization: Explicit noise-level conditioning remains contingent: it can ease optimization and improve calibration across corruption regimes, but masked objectives may instead be organized around corruption patterns and unresolved-token counts.Its usefulness depends on whether one denoiser must cover states ranging from nearly clean to nearly destroyed.
  • Score and ratio parameterizations: Score-based training can collapse to mean-parameterized denoising when reverse-process compatibility is enforced, while unconstrained scores may become inconsistent with forward dynamics.This makes parameter constraints important for preserving a valid reverse process rather than treating score prediction as an unconstrained objective.
  • Score and ratio parameterizations: Ratio matching retains links to continuous-time likelihood and reverse-process theory, but denoising-compatible implementations such as CEDD address the computational fragility of naive large-vocabulary parameterizations.The practical distinction is between elegant but brittle unconstrained scores and scalable ratio parameters trained through denoising cross-entropy.

7 Inference Algorithms and Efficiency

Discrete diffusion inference is a policy layer over a fixed denoiser, exposing choices over time schedules, token selection, block structure, guidance, and acceleration. These choices support parallel iterative refinement but create trade-offs among dependency correctness, quality, validity, likelihood, and runtime.

  • Inference as a policy layer: Inference exposes composable decisions over time points, token reveals or revisions, block structure, external guidance, and Transformer computation.These sampler decisions affect quality, validity, and runtime.
  • Ancestral sampling and step schedules: Step schedules reduce effective computation through shortened chains, selective denoiser calls, higher-order updates, and planned trajectories.Discrete Non-Markov Diffusion Models call the denoiser only when tokens can change, while other methods optimize or dilate denoising schedules.
  • Efficiency and acceleration: Acceleration methods distinguish fewer nominal steps from fewer denoiser calls, lower per-call numerical error, or avoiding unproductive trajectory segments.Distillation trains students, samplers, or consistency structures for fewer refinement steps, while some denoising steps need not use full-sequence Transformer calls.
  • Confidence-based remasking: Confidence-based token selection enables parallel updates, while remasking and backtracking permit iterative correction of unreliable earlier decisions.Entropy, predictive-distribution stability, and other signals determine which positions to decode, reopen, or update.
  • Confidence-based remasking: Parallel decoding faces dependency error because jointly updating mutually dependent tokens can violate the conditional assumptions underlying parallel generation.Parallelize-and-merge methods therefore require careful merging and verification.

8 Scaling and System Considerations

Section 8 shows that discrete diffusion’s scalability depends on backbone design, task- and corruption-sensitive scaling behavior, inference-time computation, and deployment constraints. Bidirectional architectures support full-context denoising, while parallel updates can improve hardware utilization but introduce distinctive length-budget and step-scheduling trade-offs.

  • Backbone architectures: Bidirectional Transformers naturally support diffusion’s progressive full-context denoising by using global attention over corrupted or masked sequences.The diffusion objective is architecture-agnostic, leaving room for alternatives such as SSMs and recurrent architectures.
  • Backbone architectures: Global bidirectional attention exposes both negative and positive relative-position offsets, changing positional-encoding behavior and potentially improving length generalization.AR training exposes only non-negative offsets, whereas diffusion training provides a more symmetric offset distribution.
  • Scaling questions: DLMs can scale competitively with AR models, but compute-optimal token-to-parameter ratios may differ by noise type and data- versus compute-bound regime.LLaDA reports competitiveness at the 8B scale under matched compute budgets, while uniform diffusion may favor a more parameter-heavy, data-light profile in data-bound settings.
  • Inference-time trade-offs: DLM inference trades generation quality against denoising steps and test-time compute, with additional steps helping complex logical and planning tasks.Unlike AR decoding, inference depth need not correspond one-to-one with output length.
  • Inference-time trade-offs: Fixed initial mask budgets and latent intermediate states make generation length part of reasoning-time computation rather than a universally beneficial resource.Masked DLMs often begin with a fixed number of mask tokens instead of dynamically determining length through <eos>.
  • Inference-time trade-offs: Uniform step compression can reduce quality, motivating adaptive schedules such as ASS, while corruption strategies may need to match the downstream task regime.Early denoising stages are typically lower-confidence and more uncertainty-sensitive, and current evidence does not establish a universal best corruption strategy.
  • Deployment considerations: Parallel denoising lets DDMs update many tokens simultaneously, enabling better use of GPUs and TPUs in some serving settings and potentially high throughput.Mercury reports very high throughput for diffusion-based coding models, though the supplied passage does not provide the numerical value.

9 Applications … 9.3 Multimodal Foundation Models (Text + Vision/Audio)

The applications section applies a common corruption–denoising framework across engineered-token and naturally discrete domains, with text and code established while other domains vary in maturity. Multimodal diffusion replaces conventional autoregressive backbones with bidirectional masked denoising, raising design questions around token fusion, corruption schedules, and error correction.

  • 9 Applications: Discrete diffusion applications span engineered-token domains—including language, code, multimodal models, and tokenized media—and naturally discrete domains such as proteins, genomics, molecules, graphs, planning, and agents.Text, code, and tokenized media are described as well established, whereas proteins, genomics, and molecules/graphs remain active but more specialized.
  • 9.1 Language Modeling and Text Generation: Text-generation research covers unconditional, sequence-to-sequence, controllable, rewriting, summarization, augmentation, instruction-following, and emerging reasoning-oriented applications.Iterative refinement allows control signals to be injected at multiple stages for attributes, conversation structure, commonsense knowledge, synthetic data, and text editing.
  • 9.1 Language Modeling and Text Generation: Language modeling has progressed from embedding-space and simplex diffusion to masked diffusion LMs and large-scale diffusion LLMs.Reported advantages include global revision, bidirectional conditioning, controllable refinement, and robustness to local generation errors; challenges include multi-step latency and weaker streaming behavior.
  • 9.2 Code: Infilling, Repair, and Structured Constraints: Code diffusion commonly combines absorbing masking over code-subword vocabularies, bidirectional Transformer denoisers, reweighted denoising objectives, and samplers with syntactic or test-based constraints.Its iterative trajectory supports strict syntactic validity, mathematical-equation discovery, and dynamic validation such as unit-test generation.
  • 9.3 Multimodal Foundation Models (Text + Vision/Audio): Diffusion multimodal large language models jointly handle text with images, video, or audio using bidirectional masked denoising instead of conventional autoregressive backbones.This enables understanding and generation through one unified decoder.
  • 9.3 Multimodal Foundation Models (Text + Vision/Audio): Multimodal systems either process shared discrete tokens with a single denoiser or retain separate modality encoders connected by cross-attention.The unified-token strategy is exemplified by Show-o, Unified Multimodal Discrete Diffusion, MMaDA, Muddit, and Lumina-DiMOO.
  • 9.3 Multimodal Foundation Models (Text + Vision/Audio): Multimodal corruption schedules must account for unequal information content across streams, because identical masking ratios can waste capacity on image inpainting and severely degrade text.Architectures may share one timestep across modalities through separate noise functions or assign distinct timesteps to each stream; masking distributions also shape learned visual emphasis.
  • 9.3 Multimodal Foundation Models (Text + Vision/Audio): dMLLMs can propagate early mistakes through later denoising iterations, while context tokens may anchor repetitive or degenerate generations.Corrective methods reintroduce masks mid-process so earlier errors can be revised.

9.4 Tokenized Media Generation (Images, Audio, Video)

Tokenized media generation compresses images, audio, and video into discrete codes, applies categorical diffusion in token space, and decodes the predicted codes back into signals. This framework supports parallel generation and flexible editing, while extending naturally across visual, audio, speech, and structured discrete outputs.

  • The shared pipeline: The canonical pipeline is continuous data → tokenizer or codec → discrete diffusion in token space → decoder.VQ-Diffusion established this recipe by replacing Gaussian transitions with categorical transitions over VQ-VAE codes.
  • Images and videos: Tokenized discrete diffusion is especially effective for image editing, infilling, and constrained generation.Masked generative transformers enable rapid editing through discrete token manipulation.
  • Audio and speech: Neural codecs, commonly using residual vector quantization, represent audio as multiple parallel discrete-token streams per frame.Diffusion over these tokens supports parallel decoding, infilling, and editing while fitting codec-based speech and music architectures.
  • Strengths and limitations: Parallel decoding, remasking-based editing, and reusable text or multimodal backbones enable inpainting, outpainting, and localized editing without retraining.The workflow re-masks and re-decodes selected token regions under desired conditioning.

9.5 Proteins and Biological Sequences · 9.6 Genomics (DNA/RNA) and Regulatory Sequences · 9.7 Molecules and Graph Generation

Across proteins, nucleic acids, and molecular graphs, discrete diffusion adapts categorical denoising to domain-specific structure, constraints, and objectives. The sections emphasize global refinement, structural conditioning, physical or chemical validity, and evaluation against domain-relevant signals.

  • 9.5 Proteins and Biological Sequences: Proteins map naturally to discrete diffusion because their twenty-letter amino-acid alphabet forms a small categorical space with biochemical meaning.Masked and absorbing-state diffusion is commonly adapted for conditional design under stability, motif, and functional constraints.
  • 9.5 Proteins and Biological Sequences: Protein applications span functional design, membrane proteins, and shrinking proteins while preserving function.The cited work extends beyond core protein generation to these specialized design problems.
  • 9.5 Proteins and Biological Sequences: Protein generation increasingly conditions on three-dimensional structure through inverse folding, structural priors, geometric constraints, and backbone inpainting.These approaches include discrete flow matching, mask-prior and denoising strategies, and geometric conditioning.
  • 9.5 Proteins and Biological Sequences: Protein evaluation combines sequence and structure proxies with wet-lab measurements of stability, binding, and function when available.Common proxies include sequence perplexity, native-sequence recovery, motif accuracy, scTM, and pLDDT.
  • 9.6 Genomics (DNA/RNA) and Regulatory Sequences: Nucleic acids use a four-nucleotide alphabet, yet regulatory grammar, splicing motifs, and secondary structure create long-range dependencies across hundreds or thousands of positions.Global bidirectional refinement can coordinate regulatory elements across the sequence rather than committing left to right.
  • 9.6 Genomics (DNA/RNA) and Regulatory Sequences: Discrete diffusion supports RNA secondary-structure prediction, three-dimensional RNA inverse folding, and joint sequence–structure generation for nucleic-acid and protein complexes.The cited methods use discrete base-pairing representations, hyperbolic diffusion, and SE(3)-equivariant diffusion.
  • 9.7 Molecules and Graph Generation: Molecular graphs are intrinsically discrete, with categorical atom and bond attributes, making transition-matrix diffusion with graph denoising a natural formulation.DiGress is presented as the canonical instantiation, independently corrupting node and edge categories.
  • 9.7 Molecules and Graph Generation: Molecular graph generation must enforce chemical validity, including valence, connectivity, property constraints, and atom–bond consistency.Guidance and constrained sampling address validity, while MolDiff targets inconsistencies caused by independently generating atoms and bonds; related applications include retrosynthesis, spectrum-conditioned generation, and inorganic-material and crystal generation.

9.8 Planning, Agents, and Decision Making · 9.9 Tabular Data and Imputation

Discrete diffusion extends planning and decision making through iterative global refinement of structured discrete plans, trajectories, tool outputs, and combinatorial solutions. In tabular data, it unifies categorical and numerical handling while framing imputation as masked conditional generation.

  • 9.8 Planning, Agents, and Decision Making: Discrete diffusion applications span language agents, embodied control, multimodal reasoning, and combinatorial optimization through planning, structured action generation, and constraint-aware decision making.These applications use iterative refinement in discrete spaces across multiple planning and agent settings.
  • 9.8 Planning, Agents, and Decision Making: Unlike autoregressive agents, diffusion agents iteratively refine complete plans or reasoning traces, allowing later context to revise earlier decisions and reduce error accumulation.This supports implicit backtracking and correction in language-based, multimodal, embodied, and vision-language agents.
  • 9.8 Planning, Agents, and Decision Making: In offline reinforcement learning, discrete diffusion models tokenize states, actions, and optional rewards into trajectories, then generate or guide samples toward high-reward behaviors.The model learns a generative distribution over trajectories rather than directly learning a policy.
  • 9.8 Planning, Agents, and Decision Making: For tool use, diffusion directly generates categorical tokens for JSON outputs or function calls while jointly refining structured fields under schema constraints.Global context supports coherent refinement across all fields in a structured output.
  • 9.8 Planning, Agents, and Decision Making: For TSP, scheduling, and graph construction, discrete diffusion treats candidate solutions as tokens and iteratively refines them as a stochastic alternative to classical search.The process combines local-search-like refinement with global context and learned transitions.
  • 9.8 Planning, Agents, and Decision Making: Across planning domains, discrete diffusion promises global consistency, flexible conditioning, and constraint compatibility, but depends on tokenization, corruption processes, and inference strategies.These methods are complementary to, rather than replacements for, autoregressive and classical planning methods.
  • 9.9 Tabular Data and Imputation: Tabular diffusion must model rows containing both categorical and numerical fields, using small per-column vocabularies for categories and discretization into bins for numerical values.Categorical corruption follows the same discrete forward process used for tokens.
  • 9.9 Tabular Data and Imputation: Missing-value imputation maps directly to masked discrete diffusion: observed entries provide context, missing entries are masked, and the reverse process fills them conditionally.Imputation and generation require no special-purpose architecture and differ only in masking patterns.

9.10 Other Case Studies

Discrete diffusion models extend beyond text and media generation to diverse discrete or compact-codebook case studies, including layouts, 3D assets, creative media, recommendation, retrieval, and scientific domains. The paper recommends standardized reporting of tokenization, corruption, objectives, architectures, and sampling to improve cross-domain comparison.

  • Layout and document generation: Layout generation models element classes and bounding-box bins jointly as categorical variables under masked or absorbing transitions.This includes document, slide, GUI, and scene layouts.
  • Sketches, fonts, and stylized media: Structured creative tasks use discrete diffusion for sketch-conditioned inpainting, sketch-and-context generation, glyph synthesis, font creation, and Chinese calligraphy.These tasks involve small, well-defined components.
  • Recommendation and information retrieval: Recommendation and retrieval methods model user–item histories as discrete sequences using masked-denoising approaches, including fuzzy diffusion, LLaDA-Rec, and DiffGRM.The case studies recast sequential and generative recommendation within discrete state-space diffusion.
  • A checklist for new case studies: The proposed case-study checklist covers tokenization, corruption, training objectives, model architecture, and sampling procedures for consistent cross-domain comparison.It asks authors to report codebook size, granularity, pretraining, transition type, noise schedule, objectives, auxiliary modalities, sampling steps, and remasking strategy; the checklist is a recommendation rather than an established standard.

10 Evaluation and Benchmarks

Discrete diffusion evaluation requires metrics tailored to corruption–denoising generation rather than direct transfer of autoregressive conventions. Comprehensive protocols should align likelihood interpretation, domain-specific quality, self-correction, efficiency frontiers, and editing fidelity.

  • Likelihood, perplexity, and calibration: Diffusion likelihoods and AR perplexities are not directly comparable because objectives, tokenization, normalization, timestep estimation, and inference procedures differ.Reports should distinguish exact likelihoods, ELBOs, upper bounds, and surrogate denoising losses while matching preprocessing and context length across models.
  • Generation quality: Quality evaluation should use task- and modality-specific metrics because overlap measures, FID, and individual distributional metrics miss diverse outputs and distinct encode–generate–decode failures.Protocols should separately assess tokenizer reconstruction, downstream generation quality, and full-pipeline reconstruction consistency.
  • Generation quality: Diffusion-specific evaluation should measure attribution and iterative self-correction, since arbitrary generation order breaks standard AR watermarking assumptions and bidirectional revision supports global constraint satisfaction.These capabilities are structurally unavailable to left-to-right AR models without explicit rollback.
  • Speed, cost, and efficiency: Efficiency reports should include forward passes, wall-clock latency, throughput, memory, hardware, batch size, sequence length, and acceleration settings rather than denoising step count alone.Quality–latency frontier plots are preferred because diffusion exposes a continuous quality-speed tradeoff, while guidance and verification can require multiple model calls.
  • Editing, infilling, and conditional fidelity: Editing benchmarks assess infilling consistency, minimal-change editing, and constraint adherence across modalities, including code infilling measured by pass@k on span-masked Python functions.Editing requires outputs that remain high-quality and context-consistent while minimally disturbing unmodified regions.

11 Discussion: Optimization, Controllability, and Trustworthiness

Discrete diffusion is framed as iterative global refinement and a stochastic optimization process, with revisable denoising trajectories that contrast with autoregressive commitment. Its advantages are task-dependent, while trustworthiness opportunities and risks—including uncertainty signaling, hallucination hypotheses, and safety attack surfaces—remain incompletely characterized.

  • Optimization and controllability: Each denoising step updates many positions using the entire sequence, enabling coarse-to-fine planning that fixes global structure before local detail.This permits later revisions to earlier plan or reasoning content, unlike left-to-right autoregressive commitment.
  • Optimization and controllability: Remasking and confidence-based decoding revisit uncertain positions, allowing mistakes to be corrected as neighboring tokens provide more context.The stated practical benefits are improved global coherence and constraint satisfaction.
  • Optimization and controllability: The reverse chain is proposed, interpretively rather than as an established result, as a stochastic optimizer that concentrates probability on high-likelihood or high-reward regions.The framing connects discrete diffusion to combinatorial search and offline black-box optimization.
  • Optimization and controllability: Diffusion is not uniformly preferable: autoregressive and hybrid methods remain advantageous for streaming, incremental interaction, mature KV-cache deployment, editing, infilling, and mixed generation workflows.The choice should match task structure, including latency, revision needs, constraint density, and streaming requirements.
  • Trustworthiness: Claims that bidirectional context and revision reduce hallucination remain plausible but unestablished, because evidence is thin and parallel decoding may introduce jointly sampled-token failures.The evidence is also confounded by differences in model scale and training data.
  • Trustworthiness: Per-token, per-step confidence signals could support abstention, clarification, retrieval, or verification, while guidance and hard constraints enable safety steering but introduce new attack surfaces.The confidence signals require calibration before becoming actionable uncertainty estimates.

12 Future Directions

Future work centers on understanding scaling and standardizing evaluation, closing discrete diffusion’s gaps in in-context learning, caching, and streaming, designing unified token spaces, and strengthening theoretical foundations.

  • 12.1 Scaling laws and benchmark standardization: Scaling laws for discrete diffusion remain less understood than for autoregressive models, including trade-offs among denoising steps, parameters, training tokens, and compute-optimal training.Open questions include whether diffusion language models exhibit comparable large-data scaling laws and whether smaller models trained longer remain compute-optimal across domains and scales.
  • 12.1 Scaling laws and benchmark standardization: Benchmark standardization is prerequisite for interpreting diffusion’s quality-compute frontier and settling scaling questions through minimum reporting of steps, latency, hardware, decoding, batch, and prompt lengths.Single-number comparisons can misrepresent systems because diffusion decoding exposes a quality-compute frontier rather than one operating point.
  • 12.2 In-context learning, caching, and streaming: Few-shot in-context learning is weaker in current diffusion language models than in comparable autoregressive models, potentially because of objective mismatch, prompt-response interference, or insufficient scale and instruction tuning.The reasons for this gap remain poorly understood, motivating directions that address the denoising objective, bidirectional refinement, model scale, and instruction tuning.
  • 12.2 In-context learning, caching, and streaming: Bidirectional attention and whole-canvas updates hinder append-only KV caching and streaming, although block-wise reuse, incremental unmasking caches, speculative refinement, and confidence-gated freezing enable partial reuse.No token is final until reverse diffusion completes, complicating streaming output; a clean analogue of the autoregressive KV cache remains unresolved.
  • 12.3 Unified token spaces: Unified token spaces could span text, image, audio, and video tokens or coordinate modality-specific codebooks through a common diffusion process, while raising granularity, topology, and allocation questions.The design challenge follows from treating the discrete state space as a first-class axis; unified spaces promise simpler architectures and cross-modal transfer.
  • 12.4 Theoretical foundations: Discrete diffusion theory remains incomplete on convergence and identifiability, despite guarantees and error bounds for several discrete samplers and unresolved bias and parameterization questions.Sharper theory could guide schedule and transition-matrix design and help anticipate failure modes rather than discovering them empirically.

13 Conclusion

The conclusion frames discrete diffusion around co-adapting tokenization, corruption, learning, and sampling as an inseparable system. It highlights parallel refinement and constrained generation while anticipating coexistence with autoregressive models through hybrid systems.

  • Unified framework: The survey treats discrete state-space construction, especially tokenization, as a first-class design axis and organizes formulations around four components.The four components are the corruption operator, denoiser parameterization, training objective, and sampler.
  • Design principles: Effective discrete diffusion designs co-adapt state-space design, corruption, learning objective, and sampling algorithm to data structure and task demands.The conclusion presents these choices as mutually constraining rather than independent components.
  • Strengths: Discrete diffusion is particularly suited to parallel refinement, infilling and editing, and constraint-aware generation across domains.
  • Future directions: Autoregressive and diffusion paradigms are expected to coexist, with hybrid systems combining AR planning and diffusion refinement or diffusion editing with AR backbones.
  • Future directions: Open problems include scaling laws, in-context learning, caching, unified token spaces, and theoretical foundations.The conclusion characterizes these directions as concrete and tractable.

Broader Impact

The survey highlights dual-use, safety, privacy, and accessibility concerns arising from discrete diffusion methods across scientific generation, language modeling, synthetic data, and broader deployment. It presents mitigations and reporting priorities without claiming to resolve these concerns.

  • Dual use in scientific generation: Scientific generation lowers barriers to designing biological and chemical sequences, while conditioning and guidance can be redirected toward harmful targets.Constraint-aware methods can support both hard validity and property constraints, making their implications relevant in both beneficial and harmful directions.
  • Safety of diffusion language models: Discrete diffusion language models may have a different safety surface from autoregressive models because bidirectional, multi-step, arbitrary-order decoding can enlarge jailbreak risks.The survey calls for diffusion-specific, order-varying attack evaluations, membership-inference and privacy tests, and assessment of watermarking schemes.
  • Synthetic-data privacy: Synthetic-table fidelity can leak information about real records, creating tension among fidelity, privacy, and downstream utility.The survey recommends reporting membership-inference or distance-to-closest-record analyses alongside fidelity and utility, rather than optimizing fidelity alone.
  • Accessibility and concentration of capability: Consolidating techniques into a single design reference is intended to broaden access, reproducibility, and scrutiny while also extending access to dual-use capabilities.The survey views wider, better-documented access and proposed reporting and evaluation standards as net positive, while recognizing associated risks.
Loading 2607.13431v1…