Entropic Time Schedulers for Generative Diffusion Models

1 Introduction

• Noise scheduling affects both temporal weighting during training and discretization during inference, so poor choices can undersample some windows and oversample others.The introduction notes that generative decisions may cluster in critical time windows.Source: §1 · p. 1
• Entropy-based time makes each point contribute equally to generation and is invariant to the original time parameterization.The paper also gives an exact, tractable formula relating these quantities to empirical EDM and DDPM losses for defining schedules for trained networks.Source: §1 · p. 2
• Prior data-dependent time-warping for discrete-token diffusion required specialized architectures trained with cross-entropy rather than standard denoising score matching.The paper presents its approach as tractable for continuous generative diffusion models.Source: §1 · p. 2
• Figure 1 illustrates the same SDE and its conditional entropy in standard and entropic time.Source: Figure 1 · p. 2

2 Related work

• The paper frames its contribution as an information-theoretic approach to sampling schedules, distinct from work that learns or empirically determines schedules.The review covers solver alignment, generation-step categories, spectral analysis, and joint step-and-architecture optimization.Source: §2 · p. 2
• The closest cited work uses cross-entropy to derive time warping for diffusion language models, whereas this paper studies standard diffusion models without cross-entropy.The paper also says that work's entropy expression assumes conditional independence of tokens given the noisy state.Source: §2 · p. 2
• Unlike Li et al. (2025), which studies conditional entropy between consecutive steps on a fixed grid, this paper uses entropy between the current step and time zero, invariant to time and discretization changes.The paper says exact formulas apply to generative diffusion trained with denoising score matching in continuous and discrete regimes.Source: §2 · p. 3

3 Background on score-matching generative diffusion

• A forward SDE injects noise into a target data distribution, with drift and volatility governing the process.The marginal densities are obtained from the Fokker–Planck equation.Source: §3 · p. 3
• Generation starts from a sufficiently noisy state and reverses the dynamics to recover samples from the data distribution.The reverse process uses the reverse solution kernel, interpreted as the optimal denoising distribution.Source: §3 · p. 3
• The data-dependent component of reverse dynamics is the score, expressed as an expectation over the optimal denoising distribution.Source: §3 · p. 3
• In practical diffusion models, a deep network approximates the score by minimizing an upper bound on quadratic score-matching loss.The denoising score-matching objective differs from score matching by a constant, so their gradients and optima agree; the former avoids sampling from the unknown optimal denoiser.Source: §3 · p. 3

4 Optimal sampling schedule as a change of time

• For EDM with normally distributed data of variance c^2, Sabour et al. (2024) give an analytical KL-minimizing ODE sampling schedule that is also optimal for deterministic DDIM.The paper says DDIM's invariance under time change makes this schedule suitable for comparing time parameterizations.Source: §4 · p. 4
• The optimal schedule depends on the data distribution and can be viewed as a time transformation that makes sampling linear in the new time.Source: §4 · p. 4

4.1 Change of time

• A proper time change is continuous and strictly increasing, and transforms an SDE's drift and volatility according to the time map.Source: §4.1 · p. 4
• Two SDEs are equivalent up to time change when a proper time map relates their dynamics; their forward kernels remain the same under that reparameterization.With the stated normalization, the time change is unique if it exists.Source: §4.1 · p. 4
• Entropic time requires a dataset defining the initial distribution, held fixed when comparing time parameterizations of an SDE.Source: §4.1 · p. 4

5 Entropic time schedules

• Entropic time uses conditional entropy to define a clock in which information about the source data changes at a constant rate.The section also introduces information transfer as the prior entropy minus the conditional entropy of x0 given xt.Source: §5 · p. 5

5.1 Characterizing the conditional entropy

• Conditional-entropy change can be computed through entropy production, whose dependence on the data distribution is mediated by the score function.The section derives the time derivative from the entropy of the noisy state and the forward conditional distribution.Source: §5.1 · pp. 5–6

5.2 Estimating the entropy rate from the training loss

• The training loss provides a tractable estimate of conditional-entropy production, which upper-bounds the true value by a gap determined by learned-versus-true score disagreement.The conditional-entropy production is related to the gap between explicit and denoising score-matching losses; using the learned score makes the entropy estimate approximate.Source: §5.2 · p. 6

5.3 The entropic and rescaled entropic times

• Conditional and rescaled entropy define proper time changes invariant to the SDE’s initial parameterization; equivalent SDEs have the same drift and noise in these clocks.Rescaling addresses the conditional entropy’s negative-infinite value at time zero for continuous data, which makes its overall curve arbitrary.Source: §5.3 · p. 7

5.4 Spectral rescaled entropic time

• Spectral rescaled entropy weights Fourier directions by their amplitudes after setting each direction’s final rescaled entropy to one.The preceding theorems still hold frequency by frequency; Figure 2 shows an example across radial frequencies for ImageNet-64’s red channel.Source: §5.4 · p. 8, Figure 2 · p. 8

6.1 One-dimensional experiments

• For discrete data, entropic time substantially outperforms the other schedules; for overlapping Gaussian mixtures, rescaled entropic time performs best.When Gaussian variances are small relative to the distances between components, entropic time performs better.Source: §6.1 · pp. 8–9

6 Experiments

• The experiments assess few-step generation using analytic scores and KL divergence for discrete data and Gaussian mixtures, alongside FID and FD-DINOv2 for trained models.The trained-model comparisons include CIFAR-10, FFHQ, and ImageNet.Source: §6.1 · p. 8, §6 · p. 8

6.2 CIFAR10, FFHQ, and ImageNet

• Experiments compare standard, entropic, rescaled entropic, and spectral rescaled entropic schedules on trained EDM and EDM2 models across CIFAR-10, FFHQ, and ImageNet.For stochastic DDIM, spectral rescaled entropy performs better than rescaled entropy; for deterministic DDIM, their difference is negligible.Source: §6 · p. 8, §6.2 · p. 9
• The rescaled entropic schedule produces lower-brightness images than EDM in same-seed image comparisons using deterministic DDIM at 8, 16, 32, and 64 NFEs.The supplied figure caption identifies the settings for the visual comparison; it does not quantify image-quality scores.Source: §6.2 · p. 9, Figure 4 · p. 10

6.3 Limitations

• Entropic schedules show benefits across datasets and fast-sampling methods, but those benefits are observed specifically for first-order solvers.The authors report worse results than the EDM schedule with second-order solvers.Source: §6.3 · p. 9

7 Conclusions and future work

• The authors conjecture that conditional entropy gives an optimal schedule for discrete generative tasks, but state that they lack a theoretical proof.Their toy examples showed promise, and empirically aligning entropic time with discrete diffusion is proposed as future work.Source: §7 · p. 10
• Entropic time could identify informative supervision stages in distillation and reduce redundancy when transferring knowledge to students or consistency models.These are proposed applications, not reported experimental results.Source: §7 · p. 10
• The authors propose entropic time as a candidate training schedule aligned with information flow and envision more efficient, adaptive training across modalities.They frame replacing dataset-specific optimization as a possible future outcome.Source: §7 · p. 10
• Second-order solvers with lookahead steps may require a different definition of information transfer, motivating solver-tailored entropic analogues.Source: §7 · p. 10

• The checklist asks whether the abstract and introduction accurately state the paper’s claims and contributions.Source: NeurIPS Paper Checklist · p. 13
• Claims should match the theoretical and experimental results and reflect how far the findings can be expected to generalize.Source: NeurIPS Paper Checklist · p. 13
• Important assumptions and limitations should be stated, while aspirational goals may be included when clearly identified as unattained.Source: NeurIPS Paper Checklist · p. 13

2. Limitations

• Image metrics were calculated only once, so the reported values do not capture variation across repeated evaluations.The checklist response says this is why the paper does not report statistical-significance information.Source: §2 · p. 15

A Broader impact

• More efficient generation may lower the barrier to large-scale synthetic-media misuse, including disinformation or impersonation.Source: A Broader impact · p. 20

B Entropy Production

• The entropy-production derivation uses the Fokker–Planck equation and integration by parts.Source: B Entropy Production · p. 20

• For EDM Gaussian data, deterministic DDIM’s KL-minimizing schedule is the same optimal schedule given by Sabour et al. (2024).The proof obtains a simplified DDIM update using an analytical score and identifies it with the ODE Euler update.Source: C Optimal schedule for deterministic… · pp. 20–21

• Conditional entropy and its integrated entropy-production form define proper time changes under the stated SDE and initial-data setup.The proof relies on positive entropy derivative, except for a single-point initial distribution.Source: D Proofs from section 5.3 · p. 21
• The composed time change implementing equivalence between the two SDE parameterizations is the identity map.The same identity result applies to rescaled entropic time because the noise scale is invariant under proper time changes.Source: D Proofs from section 5.3 · p. 21

E Rescaled entropy for Gaussian data

• For the EDM noise schedule and Gaussian data of variance c^2, rescaled entropic time gives the optimal ODE sampling schedule, matching equation 6.The appendix derives the linear schedule in rescaled entropic time.Source: E Rescaled entropy for Gaussian data · p. 22

F Connection with a squared error and loss

• The appendix connects conditional entropy production to squared error and expresses the time-t loss using that error.The squared error represents uncertainty about the final sample x0.Source: F Connection with a squared error and… · pp. 22–23

G Tweedie’s second-order formula

• Under a Gaussian-noise observation model, the Tweedie-formula derivation yields a general expression for the posterior variance.The appendix extends the one-dimensional derivation to arbitrary dimensions.Source: G Tweedie’s second-order formula · pp. 23–24

H.1 Algorithms

• Algorithm 3 estimates the squared-error spectral decomposition from data, enabling computation of entropic quantities in different orthonormal bases.Source: §H.1 · p. 24

H.2 One-dimensional experiments

• One-dimensional tests used equidistant steps in standard, entropic, and rescaled entropic time with stochastic DDIM on discrete point mixtures and Gaussian mixtures.Entropy estimates used squared error; the discrete and continuous cases evaluated KL divergence using different distribution-estimation procedures.Source: §H.2 · pp. 24–25
• Figure 5 plots KL divergence against generative-step count for discrete point mixtures and continuous Gaussian mixtures across time parameterizations.The supplied passages describe the comparison but do not report which schedule performs best.Source: Figure 5 · p. 25

H.3 CIFAR10, FFHQ, ImageNet

• For image benchmarks, entropy estimates used squared error at 128 EDM-schedule time points with 1,024 samples each; 128 versus 512 points showed no perceivable CIFAR10 or FFHQ curve difference.The experiments used 50,000 generated images, FID and ImageNet FD-DINOv2, and results from a single run.Source: §H.3 · pp. 25–26
• At NFE=16, DINO-optimized models gave better FID than FID-optimized models; for DINO-optimized models, FID could rise while FD-DINOv2 steadily decreased.The authors note that the two metrics assess and value different image properties.Source: §H.3 · p. 26, Table 5 · p. 27, Table 6 · p. 27
• The unrescaled entropic schedule produced blurry images and was excluded from further comparison.The appendix illustrates this with deterministic DDIM over 64 steps using EDM2-L.Source: §H.3 · p. 26, Figure 6 · p. 27
• Figure 7 compares EDM and rescaled entropic schedules with matched random seeds under deterministic DDIM at 8, 16, 32, and 64 NFEs.The appendix also provides 64-step image examples for stochastic and deterministic DDIM across EDM2 model sizes and optimization targets.Source: Figure 7 · p. 28, Figure 8 · p. 28, Figure 9 · p. 28