PaperlayerView exact revision on arXiv

Source-linked AI summary

Jumping up and down: Denoiser diffusion models for discrete ordinal data

Yair Shenfeld, Ricardo Baptista, Stefano Peluchetti

arXiv:2610.02754v1cs.LGstat.ME

TL;DR

Diffusion models for discrete ordinal data remain less developed than those for continuous or categorical data, despite the prevalence of ordered integer-valued data. JUD trains latent denoisers for bidirectional perturbations and is validated across synthetic, image, and reasoning tasks, including state-of-the-art FID on CIFAR-10 among discrete diffusion models.

  • Problem

    Diffusion models for prevalent discrete ordinal data are less developed than those for continuous domains and categorical data, although ordinal values have a natural order that makes denoising meaningful.

    Source: §1 · pp. 1–2

  • Method

    JUD trains latent denoisers for partially noised ordinal data and uses a stochastic process that allows up-and-down perturbations to sample from the data distribution.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    JUD is validated on synthetic datasets, CIFAR-10, and Sudoku completion, and achieves state-of-the-art FID on CIFAR-10 among discrete diffusion models.

    Source: §1 · p. 2, §4 · p. 9

  • Takeaways & Limitations

    Theorem 2.1 establishes that learning latent denoisers suffices to sample from the data distribution.

    Source: §1 · p. 2, §4 · p. 9

    The method currently requires a high number of neural function evaluations for sampling.

    Source: §5 · p. 10

Abstract

from arXiv · show

Diffusion models are highly developed in continuous spaces for image and video domains. Recently, major advances have been made for discrete diffusion models for categorical data, specifically in the language domain. In contrast, diffusion models for discrete integer-valued data are less developed, despite the prevalence of this modality, ranging from images and music to gene counts. We introduce Jumping Up and Down (JUD)---a new family of denoiser-based diffusion models for discrete ordinal data. This is the first family of diffusion models for ordinal data which centers around training denoisers, which at the same time allows for bi-directional (up and down) perturbations of the data. The simplicity of the training objective, combined with the flexibility of bi-directional perturbations, leads us to obtain competitive results across different data modalities.

1 INTRODUCTION

Ordinal integer data is common but less developed in diffusion modeling than categorical data, even though its natural order makes denoising meaningful. JUD uses denoiser-based models with up-and-down perturbations and reports competitive results across synthetic and real-world tasks.

1 INTRODUCTION

  • Ordinal data is common in images, music, tabular data, and scientific applications, yet diffusion theory and practice for it are still developing.Unlike categorical data, ordinal values have a natural order, placing the modality between categorical and continuous data.Source: §1 · p. 1
  • Ordinal denoisers can be trained with Bregman losses, including mean-square error, while categorical diffusion commonly uses discrete scores or generative-process rates.The paper also notes that denoiser-based techniques from continuous diffusion could in principle be adapted to the ordinal setting.Source: §1 · p. 2
  • Up-only or down-only perturbations can leave a sampler unable to undo trajectory errors, motivating perturbations that allow moves in both directions.The introduction contrasts the Binomial-Poisson process with one-directional Binomial and Poisson processes.Source: §1 · p. 2
  • JUD trains latent denoisers to predict partially noised data, which Theorem 2.1 shows suffice for sampling through a process with up-and-down moves.The framework is presented for discrete ordinal data on N^D and Z^D.Source: §1 · p. 2
  • JUD is validated on synthetic data, CIFAR-10, and Sudoku completion, with state-of-the-art FID on CIFAR-10 among discrete diffusion models.The paper also reports competitive results across these data modalities.Source: §1 · p. 2
  • For a bimodal Negative Binomial Mixture, Figure 1 reports the closest W1 agreement with the data for the proposed Binomial-Poisson process.The figure compares generated-sample histograms with the true distribution.Source: §1 · p. 2, Figure 1 · p. 3

2 JUMPING UP AND DOWN (JUD)

JUD constructs discrete-time perturbations and denoiser-based continuous-time Markov chains for ordinal data on N^D or Z^D. Its processes use latent denoisers, with training and sampling procedures based on those denoisers or marginal posteriors.

2 JUMPING UP AND DOWN (JUD)

  • JUD uses Poisson or Binomial noise for ordinal data and constructs continuous-time Markov chains whose marginals match the resulting perturbations.The perturbations keep values in N^D or Z^D, respectively.Source: §2 · p. 4

2.1 THE BINOMIAL-POISSON PROCESS AND THE POISSON-POISSON PROCESS

  • The Binomial-Poisson and Poisson-Poisson processes match their respective perturbation marginals and use latent up- and down-denoisers.The Binomial-Poisson terminal distribution is exactly Poisson, whereas the Poisson-Poisson terminal distribution is approximately Skellam; the paper reports Binomial-Poisson often more effective in experiments.Source: §2.1 · pp. 4–5
  • The latent denoisers predict the removed and added noise, and the reverse-time CTMC uses their estimates to define its transition rates.The proof combines discrete Tweedie formulas for the Binomial and Poisson distributions.Source: §2.1 · p. 5

2.2 TRAINING AND SAMPLING

  • Training can regress directly on latent noise with a Bregman divergence or train coordinatewise marginal posteriors and convert them to latent denoisers.Marginal-posterior training avoids learning the full posterior over the data given the perturbed input.Source: §2.2 · pp. 5–6
  • Sampling approximately reverses the CTMC from a Poisson initialization for Binomial-Poisson or a Skellam initialization for Poisson-Poisson, using tau-leaping or Euler methods.The Poisson initialization matches the terminal distribution exactly, while the Skellam initialization is an approximation; Figure 2 presents target and generated samples for all eight synthetic distributions.Source: §2.2 · p. 6, Figure 2 · p. 7

3 EXPERIMENTS

The experiments evaluate JUD on synthetic distributions, unconditional CIFAR-10 generation, and Sudoku completion. Results favor Binomial-Poisson on almost all tested distributions and among the compared discrete CIFAR-10 models; on Sudoku, posterior CE outperforms MSE in both settings.

3.1 SYNTHETIC EXPERIMENTS

  • Across almost all tested distributions, especially multimodal mixtures, Binomial-Poisson yields closer Wasserstein-1 agreement than alternative discrete diffusion models.Evaluation trains denoisers on 20,000 target samples and compares generated samples with independent target samples.Source: Table 1 · p. 7, §3.1 · p. 7

3.2 CIFAR-10

  • FID 2.07 on 50,000 generated images is the lowest among the discrete models compared for unconditional CIFAR-10.At 512 sampling steps, FID is 2.29. The model uses a shared DDPM++ network with EDM-style preconditioning and tau-leaping sampling.Source: Table 2 · p. 8, §3.2 · p. 8

3.3 SUDOKU

  • Posterior CE achieves higher mean exact-board accuracy than MSE in both the original and digit-shuffled Sudoku settings.Sudoku is not ordinal: permuting its symbols leaves the task invariant, so this experiment explores the framework on categorical data.Source: §3.3 · pp. 8–9, Table 3 · p. 9

4 CONCLUSION

JUD is a denoiser-based diffusion family for discrete ordinal data that permits bidirectional perturbations. Its latent denoisers support sampling, and the paper reports strong results on synthetic data and CIFAR-10 among discrete diffusion models.

4 CONCLUSION

  • JUD is a denoiser-based generative diffusion family for discrete ordinal data over N^D or Z^D, with bidirectional perturbations.The conclusion identifies this as the paper's introduced framework.Source: §4 · p. 9
  • Theorem 2.1 shows that learning latent denoisers suffices for sampling, and the two latent denoisers can theoretically be expressed in terms of each other.The conclusion also states that the data denoiser can be expressed in terms of the latent denoiser(s).Source: §4 · p. 9
  • On synthetic datasets JUD often outperforms alternative discrete diffusion models, and it achieves a state-of-the-art result among discrete diffusion models for CIFAR-10.The paper also validates the methods on Sudoku completion.Source: §4 · p. 9

5 LIMITATIONS

The limitations concern source-distribution flexibility, sampling cost, and an extra tuning parameter for the Poisson-Poisson approximation. The appendix develops the Binomial-Poisson construction, its denoiser relations, and matching forward and reverse processes.

5 LIMITATIONS

  • The method requires an easily sampled source and many neural function evaluations; Poisson-Poisson sampling also requires tuning final time τ to improve its Skellam approximation.The stated source examples are Poisson or approximately Skellam.Source: §5 · p. 10

B THE BINOMIAL-POISSON PROCESS

  • The Binomial-Poisson perturbation subtracts a Binomial latent count from the data and adds an independent Poisson count, with latent denoisers predicting those counts.The process theorem identifies latent-down and latent-up denoisers as conditional expectations of the respective latent counts.Source: B THE BINOMIAL-POISSON PROCESS · p. 16

B.1 DISCRETE TWEEDIE’S FORMULA

  • A discrete Tweedie formula relates latent-up and latent-down denoisers, allowing the data denoiser to be expressed from a single latent denoiser.The stated relation is nonlinear and non-local.Source: §B.1 · pp. 18–19

B.2 FORWARD AND BACKWARD PROCESSES

  • The forward CTMC has the Binomial-Poisson perturbation as its marginals, and its time reversal supplies a sampling process initialized from the terminal distribution.The reverse chain starts from pτ = Poisson(uτ) and has marginals pτ−s.Source: §B.2 · pp. 22–23

B.3 PROOF OF THEOREM B.1

The proof constructs the Poisson-Poisson process from a perturbation that subtracts and adds independent Poisson counts. Discrete Tweedie relations support its denoisers, and forward and reverse CTMCs reproduce the perturbation marginals.

C THE POISSON-POISSON PROCESS

  • The Poisson-Poisson perturbation subtracts an independent Poisson count and adds another, and its process is constructed using latent-up and latent-down denoisers.The denoisers are conditional expectations of the respective latent counts given the noisy observation.Source: C THE POISSON-POISSON PROCESS · p. 24, §C.1 · p. 26

C.1 DISCRETE TWEEDIE’S FORMULA

  • Discrete Tweedie’s formulas relate the latent denoisers and the data denoiser for the Poisson-Poisson perturbation.The stated relations between latent-up and latent-down denoisers are nonlinear and non-local.Source: §C.1 · pp. 26–27

C.2 FORWARD AND BACKWARD PROCESSES

  • A forward CTMC is defined to have the Poisson-Poisson perturbation as its marginals, and its time reversal provides the corresponding reverse process.The reverse chain is described with sampling time s and marginals qτ−s.Source: §C.2 · p. 29

C.3 PROOF OF THEOREM C.1

The appendix relates one-directional jump cases to the two JUD processes and derives EDM preconditioning scalings. It also supplies moment formulas and target moments for both perturbation families.

C.3 PROOF OF THEOREM C.1

  • For the Poisson-Poisson process, the reverse chain starts from qτ and has marginals qτ−s, with rates given by the corresponding time-reversal result.The proof applies the Poisson-Poisson notation change to those rates.Source: §C.3 · p. 30

D.1 JUMPING UP ON ND OR ZD

  • When only upward jumps are allowed, the Binomial-Poisson and Poisson-Poisson versions coincide.The down denoiser is zero in this case.Source: §D.1 · p. 30

D.2 JUMPING DOWN ON ND

  • For downward-only jumps on N^D, the data denoiser is the conditional expectation of the data given the Binomial-thinned observation; this is identified with Shenfeld et al.’s Binomial Flow.The cited passage identifies Equation (90) as the Binomial Flow.Source: §D.2 · p. 30

E EDM SCALING

  • Centered affine EDM scaling sets the network input to zero mean and unit variance, then centers and scales the target and chooses the skip coefficient by minimizing residual variance.The final effective squared-error weight makes the expected initialization loss constant across σ.Source: E EDM SCALING · p. 32

E.1 EDM SCALING FOR JUD

  • For JUD, the scaling is instantiated for both perturbations using observation and target statistics, with coordinatewise i.i.d. assumptions for scalar preconditioning coefficients.The appendix lists target moments for the Binomial-Poisson perturbation in Table 5 and for the Poisson-Poisson perturbation in Table 6.Source: E EDM SCALING · p. 31, §E.1 · p. 32, Table 5 · p. 34

1. Using the mean and variance of the binomial distribution,

  • For Binomial-Poisson observations, the mean is ltµ + ut and the variance is VB(t) + ut.The target-specific moments are listed separately in Table 5.Source: §1 · p. 34, Table 5 · p. 34
  • For Poisson-Poisson observations, the mean is µ − lt + ut and the variance is v + lt + ut.The target moments are reported separately in Table 6.Source: §1 · p. 34, Table 6 · p. 35

F.1 DATA

The synthetic-data setup draws 20,000 independent samples per one-dimensional target distribution; the supplied examples include four specified count-distribution families.

F.1 DATA

  • Each one-dimensional synthetic target distribution is sampled with N = 20,000 independent draws.The passage introduces these as target distributions for the synthetic datasets.Source: §F.1 · p. 35
  • The evaluated synthetic targets include zero-inflated Poisson, beta-negative-binomial, Yule-Simon, and correlated Beta-Binomial distributions.The listed supports are 50, 100, 50, and 65, respectively; the correlated Beta-Binomial example is one-dimensional.Source: §F.1 · p. 35

F.2 TRAINING

This section specifies the models’ training setup and the sampling procedures used to evaluate them. It also describes a targeted training-time distribution and the CIFAR-10 sampling schedule.

F.2 TRAINING

  • All models use a residual-MLP backbone; training uses Adam, batch size 256, and 300 epochs, with losses varying by method.The Binomial-Poisson model uses mean-squared error on conditional-mean targets; count-fm and denoising score-entropy use KL divergence.Source: §F.2 · p. 35

F.3 SAMPLING

  • Evaluation draws 1,000 samples per method with 2,000-step reverse-time Euler discretization and reports mean and standard deviation across five trials of Wasserstein-1 distance.The sampler runs from T = 1 to T0 = 10^-4 and compares samples with a fresh batch of 1,000 held-out samples; tau-leaping gave similar results.Source: §F.3 · pp. 35–36

F.4 DETAILS ON FIGURE 1

  • Figure 1 compares methods on a shared Negative Binomial mixture target, using the same network and the reference distributions listed in Table 7.The Binomial-Poisson model uses Algorithm 1, the Poisson model trains the data denoiser, and other methods follow their published implementations.Source: §F.4 · p. 36, Table 7 · p. 36

G.1 TRAINING

  • For CIFAR-10, training uses 50,000 unlabeled images rounded to integer pixel values, with coordinatewise Poisson perturbations and a two-denoiser output.The preconditioning uses fixed scalar data moments; the loss uses centered affine preconditioning and target moments.Source: §G.1 · p. 36
  • For remaining training, times are sampled from a fixed mixture: 10% uniformly on [ε, 1 −ε] and 90% from a targeted proposal favoring noise levels where the pilot improves over the affine baseline.The pilot was trained for 5 million images with t uniform on [ε, 1 −ε]. Its EMA denoisers were evaluated on 4,096 training images at 81 times uniformly spaced in log(t/(1 −t)).Source: §G.1 · pp. 36–37

G.2 SAMPLING

  • CIFAR-10 sampling starts at t = 1 and decreases to ε = 10^-5; one network evaluation estimates the expected down- and up-jump latents for tau-leaping or Euler updates.The model uses τ = 1 and scalar schedules t = μt with μ = 120.7.Source: §G.2 · p. 37

G.3 SAMPLING ABLATIONS

The ablation compares samplers and step counts for a fixed trained model, and tests an added marginal-preserving rate correction.

G.3 SAMPLING ABLATIONS

  • The forward-plus-reverse generator correction preserves instantaneous marginals when conditional means are exact and reuses denoiser predictions without another network evaluation.The correction is added to tau-leaping with unit strength.Source: §G.3 · p. 37
  • At 256 steps, tau-leaping has substantially lower FID than Euler; it is slightly lower at 1,024 and 2,048 steps, while marginal-preserving rates worsen FID at every tested count.Table 8 reports CIFAR-10 FID across numbers of sampling steps (NFE).Source: Table 8 · p. 37, §G.3 · p. 37

H.1 TRAINING CONFIGURATION

This block describes training and posterior parameterization for digit tasks, then places the approach among prior ordinal-data diffusion methods.

H.1 TRAINING CONFIGURATION

  • The compared digit-model variants use an eight-block DiT, with shared optimizer settings within each digit encoding.Original digits use 10,000 updates with warmup then cosine decay and weight decay 0.05; shuffled digits use 20,000 updates with warmup then a constant schedule and zero weight decay.Source: §H.1 · p. 37

H.2 POSTERIOR PARAMETRIZATION

  • For each cell, the cross-entropy network’s nine logits approximate the clean-digit posterior conditioned on the entire perturbed solution and puzzle.The EDM-MSE output scalings are not applied to the logits.Source: §H.2 · p. 38

I RELATED WORKS

  • The authors characterize JUD as the first ordinal-data diffusion model to combine up-down perturbations with training only denoisers.They contrast this with prior approaches that use one-directional perturbations, lack a direct denoiser representation, or train distributional or generator-matching objectives.Source: I RELATED WORKS · p. 39

Related papers

Loading 2610.02754v1…