PaperlayerView exact revision on arXiv

Source-linked AI summary

Annealed L\'{e}vy Diffusion via Gaussian Surrogate Reverse Dynamics

Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Cheng Yuan, Ruizhe Zhang

arXiv:2610.19692v1cs.LGstat.ML

TL;DR

Gaussian diffusion has difficulty estimating scores in the tails of polynomial-tail targets, while existing heavy-tailed diffusion constructions constrain reverse dynamics or keep the stability index fixed. ALD prescribes a time-varying stable index through a marginal-first Gaussian scale mixture and studies two reverse samplers. It reports competitive results on light-tailed and manifold-structured benchmarks and stronger performance on heavy-tailed data, with a limitation in the posterior sampler's modified forward marginals.

  • Problem

    Gaussian perturbations provide weaker relative coverage of polynomial-tail targets, while prior heavy-tailed methods face nonlocal reverse formulations or fixed stability indices.

    Source: §1 · pp. 1–2

  • Method

    ALD specifies time-dependent stable perturbation marginals through a scalar Gaussian scale mixture and provides renoise and posterior reverse updates using Gaussian noise prediction.

    Source: §1 · p. 2

  • Results

    ALD matches DDPM on light-tailed and manifold benchmarks and outperforms DLPM there; on globally heavy-tailed distributions it outperforms DDPM while incurring substantially lower training costs.

    Source: §4.1 · p. 6, §5.2 · p. 9

  • Takeaways & Limitations

    Conditioning on the noise scale keeps the forward kernel Gaussian and the training structure identical to standard score matching across the evaluated settings.

    Source: §4.1 · p. 6

    ALD-posterior's accumulated-variance construction generally changes the forward marginal distributions relative to the original perturbation family.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Gaussian diffusion models offer a practical framework for generative modeling. Denoising L\'{e}vy Probabilistic Models (DLPM) introduce heavy-tailed perturbations with a fixed tail index throughout denoising. We introduce Annealed L\'{e}vy Diffusion (ALD). ALD varies the stability index $\alpha(t)$ to combine heavy-tailed exploration with Gaussian refinement. Our marginal-first formulation represents stable noise as a scalar Gaussian scale mixture. This formulation enables standard Gaussian noise-prediction training with one additional scale input. We study two sampling processes. ALD-renoise estimates the clean sample and injects fresh marginal noise. ALD-posterior preserves dependence between successive states through an accumulated conditional variance and a Gaussian bridge update. Both samplers reuse the same trained predictor with different scalar inputs. We evaluate ALD on four 2D benchmarks and an imbalanced nine-mode mixture. We also examine training efficiency and the effects of prediction perturbations on clipped Holtsmark data. Additional experiments evaluate MNIST and CelebA64 generation and CIFAR-10LT augmentation. ALD preserves local geometry on the evaluated light-tailed and manifold-structured targets.

1 Introduction

Gaussian perturbations provide tractable diffusion training but poorly cover polynomial-tail targets, while existing heavy-tailed approaches impose reverse-process or fixed-index constraints. ALD uses time-varying stable perturbations with conditional Gaussian prediction to combine heavy-tailed exploration and Gaussian refinement.

1 Introduction

  • Gaussian terminal priors and light-tailed perturbations poorly cover polynomial-tail targets, leaving tail-region scores difficult to estimate with finite data and training budgets.The scarcity of extreme samples constrains score estimation in those regions.Source: §1 · pp. 1–2
  • ALD introduces time-dependent α(t)-stable perturbations that combine heavy-tailed exploration with Gaussian refinement while retaining standard Gaussian noise-prediction training.A scalar scale input conditions the predictor, allowing reuse of standard diffusion backbones.Source: §1 · p. 2
  • Lévy SDE methods require nonlocal reverse formulations, while DLPM's stable marginal construction keeps α fixed and cannot independently anneal toward Gaussian refinement.These constraints motivate a construction that avoids requiring stable convolution closure across time.Source: §1 · p. 2
  • ALD defines each time's stable perturbation marginally through a sub-Gaussian representation rather than deriving it from discrete convolution or a Lévy SDE.This permits a prescribed α(t) schedule.Source: §1 · p. 2
  • The paper provides ALD-renoise and ALD-posterior reverse updates that share a trained predictor but differ in how they couple adjacent states.ALD-posterior retains the current-state contribution and tracks accumulated conditional noise variance.Source: §1 · p. 2

2 Related Work

Prior generative approaches differ in their noise families and reverse-process formulations. ALD instead prescribes a time-dependent stable index through a conditional Gaussian scale-mixture formulation.

2 Related Work

  • DLPM uses fixed-α stable noise in a discrete Markov chain, whereas LIM uses a Lévy SDE with fractional denoising score matching; t-EDM and t-Flow use fixed Student-t tail parameters.ALD prescribes a time-dependent stable index through a conditional Gaussian scale-mixture formulation.Source: §2 · p. 3
  • ALD uses the classical sub-Gaussian representation as its marginal-first forward-process foundation and varies α(t) within one reverse trajectory.Its two-phase annealing draws inspiration from Annealed Importance Sampling and parallel tempering.Source: §2 · p. 3

3 Preliminaries

The preliminaries review Gaussian ancestral sampling and fixed-index stable-noise diffusion, then motivate a marginal-first construction for varying the stable index.

3.1 Ancestral Sampling in the Gaussian Setting

  • DDPM defines a discrete Gaussian Markov forward chain and uses an adjacent-time Gaussian posterior for ancestral reverse sampling.The reverse mean substitutes a network estimate of the clean data while retaining the current state and adjacent-posterior variance.Source: §3.1 · p. 3

3.2 Heavy-Tailed Diffusion: DLPM

  • DLPM replaces Gaussian noise with α-stable noise and uses convolution closure to obtain Gaussian scale-mixture marginals and an ϵ-prediction objective.Its fixed-index construction does not directly extend to varying α because sums of independent stable variables with different indices are generally not stable.Source: §3.2 · pp. 3–4

3.3 Notation

  • Section 3.3 introduces a notation table summarizing symbols used throughout the paper.The supplied passage identifies the table's purpose but does not enumerate its notation.Source: §3.3 · p. 4, Table 1 · p. 4

4 Methodology

ALD defines time-varying stable-noise marginals through a scalar Gaussian scale mixture, allowing heavy-tailed exploration to transition to Gaussian refinement. It pairs this forward construction with two reverse samplers that use Gaussian noise prediction but differ in how successive states are coupled.

4.1 ALD Forward Process and Annealing Schedule

  • Directly defining time-dependent perturbation marginals lets ALD prescribe α(t) independently across time and transition from heavy-tailed exploration to Gaussian refinement.When α(t) = 2, the scale is fixed at St ≡1; σt separately controls noise magnitude.Source: §4.1 · p. 5, Figure 1 · p. 5
  • ALD matches DDPM on light-tailed and manifold heavy-tailed benchmarks, strictly outperforms DDPM on globally heavy-tailed distributions at lower training cost, and achieves higher fidelity than DLPM across evaluated settings.DLPM may exhibit heavy-tail hallucination on light-tailed and manifold heavy-tailed benchmarks.Source: §4.1 · p. 6
  • Conditioned on the sampled scale, ALD's forward kernel is Gaussian, so training uses standard Gaussian noise prediction with a scalar scale input.At α(t) = 2, St ≡1.Source: §4.1 · pp. 4, 6

4.2 Sampling Process and Training Objective

  • ALD-renoise estimates x0 and injects fresh noise at the full next-time marginal scale, without preserving local dependence between adjacent states.The paper reports that repeated full marginal re-noising can over-inject randomness, producing oscillatory or unstable finite-step trajectories and amplifying reconstruction errors.Source: §4.2 · p. 6
  • ALD-posterior uses accumulated conditional variance and a Gaussian reverse update that injects only adjacent conditional variance, no greater than Vt−∆t.This sampler changes the forward marginal distributions relative to the original perturbation family.Source: §4.2 · p. 7

A valid Gaussian transition therefore requires

  • For ALD-posterior, the predictor receives Vt/σ2t as its scale input, while the shared Gaussian noise-prediction objective uses an unweighted loss.The 10−6 offset avoids numerical overflow as the truncated scale approaches zero.Source: A valid Gaussian transition therefore… · p. 7

5 Numerical Experiments

The experiments compare ALD with DDPM and DLPM across synthetic and image-generation tasks, training-efficiency diagnostics, and downstream CIFAR-10LT augmentation. They evaluate distributional fidelity, coverage, and image metrics under matched or validation-selected settings.

5.1 Experimental Setup

  • The evaluation spans four main 2D benchmarks, an imbalanced nine-mode mixture, MNIST, CelebA64, and CIFAR-10LT, using distributional metrics and image or downstream-task measures.The paper reports sliced W2 and Gaussian-kernel MMD2 for synthetic targets, with FID, KID, and feature-space metrics additionally used for CelebA64.Source: §5 · p. 7, §5.1 · p. 8, §5.2 · p. 9

5.2 Experiment 1: Generation Accuracy on 2D Synthetic Benchmarks

  • On light-tailed and manifold benchmarks, ALD-renoise and ALD-posterior match DDPM and outperform DLPM; ALD-posterior performs best on heavy-tailed Holtsmark.Table 2 reports W2 and MMD2, where lower is better, and SHS, ranked by distance to one.Source: §5.2 · p. 9, Table 2 · p. 10, Figure 2 · p. 11
  • Under a 30:1 mixture-mass imbalance, both ALD samplers retain rare modes and local cluster structure; ALD-renoise beats DDPM and both DLPM variants, while ALD-posterior surpasses it.Table 3 reports the additional nine-mode benchmark; higher is better.Source: §5.2 · p. 10, Table 3 · p. 10

5.3 Experiment 2: Training Efficiency on Heavy-Tailed Data

  • Across tested network widths, both ALD samplers outperform DDPM; ALD-renoise leads at smaller training budgets, while ALD-posterior becomes more effective with more updates.Figure 4 plots W2 against hidden width and optimizer updates on clipped Holtsmark; lower is better.Source: §5.3 · p. 11, Figure 4 · p. 12

5.4 Experiment 3: Extension to MNIST

  • On MNIST, ALD-posterior has higher classwise kNN coverage than ALD-renoise on all ten digits and best or tied coverage on nine; DLPM leads on digit 9.Table 4 reports coverage percentages, averaging two classifier diagnostics and two sampling seeds from one generator training seed.Source: §5.4 · p. 12, Table 4 · p. 12

5.5 Experiment 4: Application to CIFAR-10LT

  • The ALD-posterior samples provide the strongest augmentation arm overall, suggesting their broader support remains useful downstream; the reported measures include CIFAR-10 test accuracy and classwise CIFAR-10 F1.Source: §5.5 · p. 12, Table 5 · p. 13, Table 6 · p. 13

5.6 Experiment 5: Performance on CelebA64

  • On CelebA64, ALD-posterior has the lowest FID and KID and highest recall, while DLPM has the highest precision, density, and coverage.At the frozen settings, ALD-posterior's FID is 2.1090, KID is 1.1419 × 10^-3, and recall is 0.5940.Source: §5.6 · p. 13

6 Discussion & Conclusion

ALD-renoise and ALD-posterior trade off training efficiency and sensitivity to prediction accuracy. ALD-posterior uses an explicit Markov forward process, but its Gaussian reverse approximation omits clean-data posterior uncertainty.

6 Discussion & Conclusion

  • ALD-renoise performs better at smaller Holtsmark training budgets, while ALD-posterior becomes more effective after more updates and is less robust to prediction perturbations.The results indicate that sampler preference depends on prediction accuracy.Source: §6 · p. 13
  • ALD-posterior defines an explicit Markov forward process with accumulated conditional variance, generally changing intermediate marginals while retaining the initial data distribution.Gaussian conditioning gives an exact adjacent posterior when the clean sample and relevant variances are known.Source: §6 · p. 13
  • During generation, the denoiser replaces the clean sample, and the resulting Gaussian reverse approximation omits clean-data posterior uncertainty from its covariance.Thus ALD-posterior is a variance-controlled sampler for an alternative forward process whose intermediate marginals differ from ALD-renoise's.Source: §6 · p. 13
  • Experiments on synthetic and image benchmarks show competitive generation quality, rare-mode coverage, and training efficiency.Table 7 reports formal CelebA64 generation using one training and sampling seed, 50,000 generated images, and 1,000 NFE per method.Source: §6 · p. 14, Table 7 · p. 14

A.1 Metrics and Reporting Conventions

The appendix specifies how sample quantiles and reported metrics are computed, including tail-focused diagnostics and classwise image coverage. These conventions constrain what each metric measures.

A.1 Metrics and Reporting Conventions

  • Nearest-rank empirical quantiles use the ordered value at index ⌈qN⌉; reported mean ± standard deviation uses population standard deviation unless stated otherwise.Table 2 instead uses sample standard deviation.Source: §A.1 · p. 14
  • W2 is a sliced distance computed over 200 random unit directions, not the full d-dimensional optimal-transport distance.MMD uses at most 1,000 points per set and a Gaussian kernel whose bandwidth is set from the median squared pairwise distance.Source: §A.1 · p. 15
  • SHS selects the coordinates with largest empirical excess kurtosis and compares directional extremes among generated samples above the real-data radial 0.95 quantile.It is conditional on the selected tail and does not estimate total tail mass or a power-law exponent.Source: §A.1 · p. 15
  • SHS directional projections use original coordinates, while standardization is used only for tail selection.The metric is undefined when the generated tail set is empty or the real-versus-real denominator is at most 10^-10.Source: §A.1 · p. 15
  • For MNIST, classwise coverage is the fraction of real images of a digit contained in generated-image feature-space balls, not the generated class frequency.For the imbalanced mixture, PRD curves are averaged across ten joint clusterings into 100 bins and 201 slopes.Source: §A.1 · p. 15

A.2 Two-Dimensional Benchmarks

The appendix specifies the 2D benchmark constructions, shared evaluation protocol, and sampler settings. ALD-posterior hyperparameters are selected on validation streams, with no corresponding extra search budget for baselines or original ALD.

A.2 Two-Dimensional Benchmarks

  • The four main 2D benchmarks are a clipped heavy-tailed cross, a noiseless Swiss Roll, a forty-component Gaussian mixture, and an independent-coordinate 3/2-stable Holtsmark surrogate.The Holtsmark benchmark is not an isotropic physical field projection.Source: §A.2 · p. 15
  • Each main dataset uses 32,000 training points per seed, and evaluation compares 5,000 generated with 5,000 shared reference points across methods.ALD and DDPM use residual MLPs; ALD additionally embeds the log scale input, while DLPM uses time conditioning, LayerNorm, and SiLU.Source: §A.2 · p. 16, Table 8 · p. 16
  • DDPM and matched DLPM apply reconstruction bounds at every reverse step, whereas ALD-posterior applies its bound to the final output.DLPM uses 199 network evaluations on the first three datasets and 74 on Holtsmark; its hyperparameters are not searched.Source: §A.2 · p. 16
  • ALD-posterior settings are jointly selected by mean validation W2, with sampling budgets of 400, 200, 400, and 50 NFE across HTC, Swiss Roll, MoG40, and Holtsmark.Baselines and original ALD receive no corresponding extra search budget, and final reference sets and checkpoints were examined in earlier comparisons.Source: §A.2 · p. 16
  • The imbalanced nine-mode benchmark uses three seeds, 32,000 training points, and 15,000 real and generated evaluation points; ALD-posterior reuses the three ALD checkpoints.Holtsmark efficiency sweeps vary hidden width from 16 to 256 at 8,000 updates and optimizer budgets from 200 to 2,000 at width 256.Source: §A.2 · p. 16

A.3 Prediction-Output Perturbation

On Holtsmark, prediction perturbations reveal different sensitivity and tail biases for the two reverse samplers. ALD-renoise's intermediate-noise improvement is not monotone robustness, while ALD-posterior performs better without added perturbation.

A.3 Prediction-Output Perturbation

  • The study reuses three Holtsmark checkpoints and perturbs each network output as ϵθ + σnoise z, with z sampled from a standard two-dimensional Gaussian.Each checkpoint is sampled three times per condition, giving nine runs.Source: §A.3 · p. 16
  • Without prediction perturbation, ALD-posterior has lower W2 and MMD2; by σnoise = 0.25, ALD-renoise has lower W2, with the gap widening at higher perturbations.At σnoise = 1, ALD-renoise's W2 and MMD2 worsen again.Source: §A.3 · p. 16, Figure 6 · p. 17
  • ALD-renoise is nearly unchanged under small perturbations but underproduces directional extremes, whereas ALD-posterior increasingly overproduces them.The comparison uses mean ± sample standard deviation over nine runs; lower is better for W2 and MMD2, while SHS is ranked by distance to one.Source: §A.3 · p. 16, Figure 6 · p. 17
  • The experiment supports an accuracy–sensitivity trade-off between the reverse rules, not uniform superiority of either sampler.The same ALD checkpoints are used, and the samplers differ only in the reverse transition.Source: §A.3 · p. 16, Figure 6 · p. 17

A.4 MNIST Protocol

The MNIST protocol specifies the training and sampling setup, classwise-coverage evaluation, and an initialization limitation for ALD-posterior.

A.4 MNIST Protocol

  • MNIST training uses 6,000 updates, batch size 128, and learning rate 2 × 10−4; DDPM, DLPM, ALD-renoise, and ALD-posterior use 10, 99, 9, and 49 evaluations.Both ALD variants use (αmin, τ, Smax) = (1.5, 0.15, 8); DLPM uses fixed α = 1.8 and scale cap 20.Source: §A.4 · p. 17
  • Classwise coverage groups generated images by two frozen classifiers without confidence filtering, then applies a kNN rule to their 128-dimensional features.Weighting classwise coverage by real class counts recovers overall recall.Source: §A.4 · p. 17
  • ALD-posterior initializes from the empirical data mean and Gaussian noise, but this omits the data-covariance contribution even at the population mean.The sampler makes N −1 network evaluations, uses the covariance in Eq. (14), and clips only the final image to [−1, 1].Source: §A.4 · p. 17

A.5 CIFAR-10LT Augmentation Protocol

The supplied protocols describe CIFAR-10LT augmentation and CelebA64 generation, including training setups, ALD sampler selection, and evaluation metrics.

A.5 CIFAR-10LT Augmentation Protocol

  • The CIFAR-10LT protocol generates 50,000 images per class-conditional generator and trains each augmentation arm on 50,000 real plus 50,000 synthetic images.The class counts range from 5,000 to 50 in the stated 100:1 imbalance; each training batch contains 64 real and 64 synthetic images.Source: §A.5 · p. 17

A.6 CelebA64 Sampling Details

  • The ALD sampler was selected from nine validation settings on the Pareto frontier, freezing (η, p) = (0.5, 1.5) at validation FID 4.5266.The settings varied η ∈ {0, 0.5, 1} and time-grid power p ∈ {1, 1.5, 2}; formal train-reference results did not change the configuration.Source: §A.6 · p. 18
  • For CelebA64, models share the architecture, training budget, generated-image count, and reference protocol, while retaining their own objectives and reverse samplers.Training uses one seed and 600,000 optimizer updates; sampling generates 50,000 images at 1,000 network evaluations per image.Source: §A.6 · p. 18
  • CelebA64 evaluation uses FID, KID, precision, recall, density, and coverage, with feature-based kNN diagnostics for the latter four metrics.These are distributional kNN diagnostics, not attribute-classification precision and recall.Source: §A.6 · p. 18
  • The validation sweep table reports scores for 5,000 generated images at 1,000 evaluations, and its values are not directly comparable with formal train-reference scores.Bold marks the frozen sampler choice, not a winner on every metric.Source: Table 9 · p. 19

Related papers

Loading 2610.19692v1…