PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Modelling With Inverse Heat Dissipation

Severi Rissanen, Markus Heinonen, Arno Solin

arXiv:2206.13397v7cs.CVcs.LGstat.ML

TL;DR

Standard diffusion models do not explicitly account for images’ multi-scale structure. This paper proposes stochastic reversal of the heat equation as a generative model; spectral analysis finds an implicit coarse-to-fine bias in standard diffusion models.

  • Problem

    Diffusion-model processes do not explicitly encode natural images’ multi-resolution structure, motivating study of that inductive bias in image generation.

    Source: §1 · p. 1

  • Method

    IHDM generates images by stochastically reversing heat-equation information loss and interprets the noisy forward solution as inference in a diffusion-like latent-variable model.

    Source: §1 · p. 2

  • Results

    Spectral analysis of natural images shows that standard diffusion models implicitly perform a different type of coarse-to-fine generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present IHDM as a first step toward reasoning about inductive biases in related generative models.

    Source: §5 · p. 9

    The authors suggest IHDM’s smoothness prior may over-regularize relative to diffusion models, and its U-Net may not be ideally suited to the method.

    Source: §5 · p. 9

Abstract

from arXiv · show

While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to-fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise as a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.

1 INTRODUCTION

The paper introduces inverse heat dissipation as a way to build multi-scale structure into diffusion-like image generation. It also examines the model’s emergent properties and analyzes coarse-to-fine generation in standard diffusion models.

1 INTRODUCTION

  • The heat equation averages images and removes fine detail while allowing arbitrary effective resolutions without reducing pixel count at each step.This contrasts with pixel-subsampling pyramids that halve resolution per step.Source: §1 · p. 1
  • IHDM stochastically reverses the heat equation to increase effective image resolution and generate plausible reconstructions from a low-dimensional prior estimated with a training-data kernel density estimate.The authors describe this as reversing the loss of image information during the forward process.Source: §1 · p. 2
  • The paper interprets a heat-equation solution with small additive noise as an inference process in a diffusion-like latent variable model.Source: §1 · p. 2
  • The authors investigate whether IHDM yields colour–shape disentanglement, smooth interpolation, simpler learned neural functions, and potential data efficiency.Source: §1 · p. 2
  • Power spectral density analysis of natural images suggests standard diffusion models implicitly perform a different kind of coarse-to-fine generation, clarifying their connections and differences from IHDM.Source: §1 · p. 2

2 METHODS

The methods use heat-equation blurring with a stochastic reverse model, and relate coarse-to-fine generation to the spectral behavior of natural images and diffusion models.

2 METHODS

  • The forward process evolves each image channel by ∂_t u=Δu with Neumann boundaries, averaging each channel to its mean and projecting asymptotically into R3.Although the heat equation is formally reversible, finite numerical precision makes its inverse ill-posed.Source: §2 · p. 2
  • A cosine-basis eigendecomposition makes heat evolution diagonal and fast, with DCT taking O(N log N); training predicts a less-blurred state from a noise-perturbed blur using MSE and skip connections.The skip connection stabilizes training, while sampling alternates deblurring updates with Gaussian noise addition.Source: §2 · p. 3, §2.1 · p. 4

2.1 GENERATIVE MODEL FORMULATION

  • Adding forward noise σ makes reverse conditionals well-defined and permits multiple paths; a Gaussian reverse Markov chain uses δ to control sampling stochasticity.The forward process treats noise as observation noise on deterministic heat evolution, rather than as a Markov chain.Source: §2.1 · pp. 3–4, Figure 3 · p. 3
  • A Gaussian kernel-density prior with variance δ^2 models the low-dimensional terminal blurred states, but its terminal loss is difficult to evaluate efficiently.The authors provide a variational upper bound to evaluate that loss without numerical integration.Source: §2.1 · pp. 4–5
  • As K→∞, L_k becomes equivalent to denoising score matching at noise σ, and δ=2σ yields Langevin sampling, but the model drops infinite-step Gaussian reverse-transition optimality guarantees.This connects the method to early score-based generative modelling (Song & Ermon, 2019; 2020).Source: §2.1 · p. 5

2.2 IMPLICIT COARSE-TO-FINE GENERATION IN DIFFUSION MODELS

  • Natural-image PSDs approximately follow 1/f^α with α≈2, so noise masks high frequencies first and diffusion models generate coarse structure before fine details; heat evolution decays frequencies explicitly, with σ setting a floor.The heat process therefore has an explicit range of effective resolutions.Source: §2.2 · p. 5, Figure 4 · p. 5

3 EXPERIMENTS

Experiments show that IHDM builds images from coarse to fine, produces promising but not state-of-the-art FID scores, and exhibits properties including colour–shape disentanglement, smooth latent interpolation, localized learned functions, and few-shot generalization.

Generative sequences

  • Generation starts from a blank image and progressively adds fine-scale structure, while stochastic bifurcations make large-scale structure first and smaller changes later.The reverse heat process redistributes the original image mass into a random image with the same average colour.Source: §3 · p. 6

Quantitative evaluation

  • 18.96 FID on CIFAR-10 for IHDM compares with 3.17 for DDPM (Ho et al., 2020) and 1.85 for current state-of-the-art methods (Sauer et al., 2022).The paper describes IHDM’s image quality as promising but not yet as good as state-of-the-art diffusion models and GANs.Source: §3 · p. 6

The importance of σ

  • With σ = 0, IHDM fails even on MNIST and produces random images; non-zero σ regularizes the unstable reverse heat process.The authors identify non-zero σ as essential for the probabilistic model to be sensible.Source: §3 · p. 6

The effect of noise δ

  • At δ = 0, sampling mainly sharpens the prior image; increasing δ above σ adds detail before images become noisy, with δ/σ ≈ 1.25–1.3 optimal.A ratio of 1.25 is reported as a good default across datasets, and the optimal ratio does not depend on the absolute value of σ.Source: §3 · p. 7, Figure 6 · p. 7

Disentangling colour and shape

  • Fixing the noise steps while changing the prior uK produces highly similar faces with different average colours in the 128×128 FFHQ model.The demonstrated model uses σB,max = 128.Source: §3 · p. 7, Figure 7 · p. 7

Smooth interpolation

  • Linear interpolation of uK and spherical interpolation of the noise yield perceptually smooth IHDM interpolations, unlike the non-smooth DDPM interpolation (Ho et al., 2020).The paper relates the smoothness to IHDM’s hierarchy of steps corresponding to different resolutions.Source: §3 · p. 7, Figure 7 · p. 7

Inductive bias on the learned function

  • Input-gradient probes show IHDM’s learned functions are circularly symmetric and localized, whereas DDPM’s dependencies are more global and complex.The authors connect this localization to the heat process’s inductive bias toward convolution-like updates.Source: §3 · pp. 7–8, Figure 8 · p. 8

Few-shot learning

  • Trained on only the first 20 MNIST digits, IHDM produces meaningful generalization, while DDPM either fails to produce convincing samples or overfits.The paper attributes the data efficiency to the generative process’s inductive bias as a prior on natural images.Source: §3 · p. 8

4 RELATED WORK

The paper situates its model among diffusion and multi-scale generative approaches, distinguishing its use of resolution increase as a foundation from prior performance-boosting additions. It also contrasts explicit heat-equation reversal for unconditional generation with blur-based inverse-problem methods.

4 RELATED WORK

  • Prior diffusion work developed score-based and probabilistic formulations, unified them through reverse SDEs, and used resolution cascades to boost performance; this model makes resolution increase foundational.The model increases resolution at every computational step.Source: §4 · p. 8
  • Resolution hierarchies have supported image generation in GANs and VAEs, while standard architectures often increase feature-map resolution through upsampling layers.Examples include progressive GAN and StyleGAN architectures (Karras et al., 2018; 2019; 2020; 2021; Sauer et al., 2022) and hierarchical VAEs (Razavi et al., 2019; Vahdat & Kautz, 2020; Child, 2021).Source: §4 · p. 8
  • Multi-scale autoregressive approaches include pixelRNN and generation from a pixelated image toward higher resolution; related work also progressively upscales resolution and bit depth.The paper identifies Reed et al. (2017) as possibly closest to its model; Menick & Kalchbrenner (2019) propose a similar resolution- and bit-depth-upscaling method.Source: §4 · p. 8
  • Unlike generative models applied to deblurring, super-resolution, and other inverse problems, this model creates an unconditional generator by directly reversing the heat equation with a simple MSE loss.Its goal is unconditional generative modelling, rather than deblurring or another inverse problem in itself.Source: §4 · p. 9
  • Concurrent blur-based diffusion work incorporates Gaussian blur into standard diffusion, while this model explicitly reverses heat dissipation and departs from the standard Markovian forward framework.Lee et al. (2022) and Daras et al. (2022) apply blur to improve diffusion performance; the paper frames their work as complementary to its investigation of blurring's inductive bias.Source: §4 · p. 9
  • Unlike cold-diffusion methods that deterministically map a blurry image to one solution, this method produces a distribution of images; other work bridges inverse heat dissipation and denoising diffusion.Bansal et al. (2022) also propose a Gaussian mixture prior for unconditional generation, while Hoogeboom & Salimans (2022) study bridging the two processes.Source: §4 · p. 9

5 DISCUSSION AND CONCLUSIONS … A.4 LIMIT OF K →∞AND CONVERGENCE TO DENOISING SCORE MATCHING AND LANGEVIN DYNAMICS

The paper presents inverse heat dissipation as a generative process with useful interpolation, disentanglement, and data-efficiency properties, and relates its infinite-step limit to denoising score matching and Langevin dynamics. The appendices detail its numerical heat solver and variational bounds, while the discussion identifies limitations and extensions to other domains.

5 DISCUSSION AND CONCLUSIONS

  • The proposed model alternates regularized deblurring with noise and exhibits smooth interpolation, latent disentanglement, and data efficiency.The authors present this as a first step toward understanding inductive biases in generative models.Source: §5 · p. 9
  • IHDM’s inductive bias appears to regularize more strongly than diffusion models, while its smoothness prior may limit fitting and its U-Net may be poorly suited to the method.The authors suggest further study of the probabilistic model, ways to loosen regularization, and architectures tailored to IHDM.Source: §5 · p. 9
  • The approach may extend wherever data have a natural coarse-graining operator, including 1D audio and graphs using the graph Laplacian, and motivates other generative sequences.The graph-Laplacian connection is described as being in the spirit of geometric deep learning (Bronstein et al., 2017).Source: §5 · p. 9

A.1 NUMERICAL SOLUTION OF THE HEAT EQUATION WITH NEUMANN BOUNDARY CONDITIONS USING THE DISCRETE COSINE TRANSFORM

  • For image heat evolution, the method uses Neumann boundary conditions and a cosine eigenbasis projected with the DCT, enabling accurate, scalable computation on regular grids.The finite image resolution imposes a cutoff frequency, making the representation finite-dimensional.Source: §A.1 · pp. 17–18

A.2 DERIVATION OF THE VARIATIONAL LOWER BOUND

  • The variational-bound appendix derives the main-text lower-bound loss by accounting for marginalized KL terms and estimating expectations with Monte Carlo samples passed through the neural network.The derivation is included for completeness and for readers less familiar with diffusion-model mathematics.Source: §A.2 · pp. 18–20

A.3 VARIATIONAL UPPER BOUND ON LK

  • A variational upper bound on LK avoids repeatedly evaluating the training-set kernel density in high dimensions by using KL distances between blurred training and test examples.The direct KDE evaluation is computationally inefficient because it requires comparing spatial locations with all blurred training examples.Source: §A.3 · p. 20

A.4 LIMIT OF K →∞AND CONVERGENCE TO DENOISING SCORE MATCHING AND LANGEVIN DYNAMICS

  • As K →∞, the loss converges to denoising score matching and sampling becomes equivalent to Langevin dynamics for the blurred-data distribution.The authors frame this as preliminary asymptotic analysis, not a derivation of the model’s actual sampling procedure, which instead takes directed steps backward through the heat equation.Source: §A.4 · pp. 20–21
  • In the asymptotic connection, the learned function estimates the score of the blurred-data density; the resulting Langevin interpretation is not the model’s practical reverse-heat sampling procedure.The authors use the connection to support preliminary statistical reasoning, not as a recipe for sampling.Source: §A.4 · p. 21

A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS · B EXPERIMENT DETAILS

In the diffusion forward process, the expected PSD of a noised image is the original image’s PSD plus the noise variance; isotropic noise can therefore drown out frequency components with lower PSD than the noise variance.

A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS

  • An individual frequency component’s PSD is the squared projection onto its DCT/DFT frequency vector, and these components compose the image PSD.Source: §A.5 · p. 21
  • The analysis considers a diffusion process that leaves the original image unscaled and adds noise ε with covariance γ^2I.Source: §A.5 · p. 22
  • The expected PSD of a noised image equals the original image’s PSD plus the noise variance.Source: §A.5 · pp. 21–22
  • Isotropic noise effectively drowns out data frequency components whose PSD is lower than the noise variance.Source: §A.5 · p. 22
  • In the 1D plot, equal-frequency components are averaged; the result remains unchanged when individual PSD values are replaced by their mean.The passage points to Appendix B.6 for a more detailed visualization.Source: §A.5 · p. 22
  • A single PSD component is distributed as the sum of a normally distributed random variable and a chi-squared variable.This uses the fact that a 1D orthonormal projection of an isotropic zero-mean Gaussian is a 1D Gaussian.Source: §A.5 · p. 22

B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS … C ADDITIONAL EXPERIMENTS

The supplied sections document the model’s architecture, training and evaluation procedures, generative-process settings, time-step schedule, and power-spectral-density calculation.

B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS

  • The transition estimator is a U-Net with residual blocks, low-resolution self-attention, 128 base channels, GroupNorm, and sinusoidal time-step embeddings.MNIST and CIFAR-10 use four residual blocks per resolution layer; other models use two.Source: §B.1 · p. 22

B.2 TRAINING TIME AND COMPUTATIONAL RESOURCES

  • Training used NVIDIA A100 GPUs, with dataset-specific iteration counts ranging from 100,000 to one million.CIFAR-10 training used 400,000 iterations; 256×256 FFHQ and AFHQ used 800,000 and 400,000, respectively, while LSUN-Churches used one million.Source: §B.2 · p. 23

B.3 FID SCORE CALCULATION

  • FID was calculated with clean-fid after bicubic resizing to 299×299, using 50,000 samples except for 256×256 datasets, which used 10,000.Reference statistics came from training sets for LSUN-Churches and CIFAR-10, and entire datasets for FFHQ and AFHQ.Source: §B.3 · p. 23

B.4 HYPERPARAMETERS RELATED TO THE NEW GENERATIVE PROCESS

  • The generative process generally used training noise σ = 0.01 and K = 200 steps, with dataset-specific exceptions and limited hyperparameter tuning.LSUN-Churches used K = 400, MNIST used 100 steps, and CIFAR-10’s δ = 0.01325 was selected by sweeping FID values.Source: §B.4 · p. 23

B.5 SCHEDULE ON tk

  • With logarithmic spacing of σB and time steps, the remaining frequency components decrease at an approximately constant rate, slowing somewhat near the end.Components become indistinguishable from training noise as the heat equation progresses.Source: §B.5 · p. 24, Figure 9 · p. 25

B.6 CALCULATION OF POWER SPECTRAL DENSITIES

  • Power spectral density is computed by squaring 2D orthogonal DCT coefficients and averaging values over equal-frequency contours for one-dimensional plots.The passage also notes that a discrete Fourier transform could be used instead.Source: §B.6 · p. 24

C.1 ROBUSTNESS TO σ AND δ

On CIFAR-10, the model is robust to σ choices except near zero, with optimal FID at approximately δ/σ = 1.3 across tested σ values. Sensitivity to δ decreases somewhat as σ increases, and samples at δ = 1.3 × σ differ little visually.

C.1 ROBUSTNESS TO σ AND δ

  • The model becomes somewhat less sensitive to δ as σ increases, while samples at δ = 1.3 × σ show little visual difference across models.Example images are shown for CIFAR-10 models trained with different σ values.Source: §C.1 · p. 26, Figure 12 · p. 26
  • On CIFAR-10, optimal FID occurs at approximately δ/σ = 1.3 across tested σ values, and the model is robust to σ choices except near zero.The tested σ values ranged from 0.005 to 0.02; nearby ratios such as δ = 1.25 × σ also work.Source: Figure 11 · p. 26, §C.1 · p. 26

C.2 DATA EFFICIENCY / FEW-SHOT GENERALIZATION · C.3 THE OPTIMAL δ WITH RESPECT TO MARGINAL LOG-LIKELIHOOD VS. FID

In a 20-example MNIST setting, IHDM generalizes to new digit shapes while the diffusion model fails or overfits. The δ optimizing NLL differs from the value optimizing FID, reflecting a trade-off the paper says may require further hyperparameter study.

C.2 DATA EFFICIENCY / FEW-SHOT GENERALIZATION

  • Trained on the first 20 MNIST digits, IHDM generalizes to new digit shapes without noticeable overfitting, unlike the diffusion model.The diffusion model either fails to produce plausible digits or completely overfits the training data.Source: §C.2 · p. 27
  • IHDM’s multi-resolution structure lets it combine learned features across scales to form new images and meaningful variation from few data points.The authors present this as an explicit way to generalize on image data.Source: §C.2 · pp. 26–27

C.3 THE OPTIMAL δ WITH RESPECT TO MARGINAL LOG-LIKELIHOOD VS. FID

  • NLL is lowest near σ = 0.01, whereas FID is lowest near δ = 0.01325, the value chosen for the model.The authors report that the optimal δ values for NLL and FID do not coincide.Source: §C.3 · p. 27
  • Without the MSE term in the KL divergences Lk, the optimal δ is σ; including the term nudges the optimum higher.The shift is small when the MSE loss is not very high.Source: §C.3 · p. 27
  • The authors say σ^2 terms dominate NLL scores in their model’s apparent regime, while changing σ could change the picture.The passage describes this as an apparent regime, not a universal result.Source: §C.3 · p. 27
  • Further study of the loss-term interplay could seek a hyperparameter regime that simultaneously optimizes marginal log-likelihood and FID.The authors frame this as a potential direction for improving the model as a marginal log-likelihood maximizer.Source: §C.3 · p. 27

C.4 EVALUATION OF THE PRIOR IN TERMS OF OVERLAP WITH THE TEST SET

Because the model draws its prior from training data, overlap between blurred train and test sets, measured by LK, serves as a prerequisite indicator of generalization. For CIFAR-10, average test-set LK decreases as σB,max increases but changes little from 24 to 32, providing little reason to expect further increases to improve generalization.

C.4 EVALUATION OF THE PRIOR IN TERMS OF OVERLAP WITH THE TEST SET

  • For CIFAR-10, average test-set LK decreases as σB,max increases but changes little from 24 to 32, suggesting larger values may not improve generalization.LK overlap is a prerequisite measure because samples from the blurred training set are used as the prior; near-zero σB,max implies little overlap and a process akin to memorizing the training set, while full averaging is expected to yield high overlap between low-dimensional blurred distributions.Source: §C.4 · p. 27

C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ

Nonzero training noise is needed to define the model sensibly and regularize reverse heat inversion. Without it, samples become random patterns; with it, the model is more robust and takes the training-data distribution into account.

C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ

  • With σ = 0, the MNIST model produces essentially random patterns.The images quickly blow up into random patterns during generation.Source: §C.5 · p. 27
  • Directly solving the exact reverse heat equation fails because estimating the extremely ill-conditioned inverse F(tK)−1 is unstable.The model learns a generic sharpening filter, so small errors in reverse steps are amplified.Source: §C.5 · p. 27
  • With nonzero σ, the model becomes more robust and is forced to take the training-data distribution into account.Training noise also acts as a regularization parameter.Source: §C.5 · p. 27

C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER · D SAMPLE VISUALIZATIONS

On CIFAR-10, the convolutional Gaussian-blur model scores worse than the DCT-based method, which the authors describe as having a minor edge. The visualizations section covers smooth interpolations, the δ parameter, feature disentanglement, and nearest-neighbour examples.

C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER

  • The sampled-Gaussian convolutional blur is a reasonable alternative, but is somewhat slower and does not directly expose frequency decay or heat-equation boundary-condition intuitions.The kernel spans (2*N −1)×(2*N −1), uses Gaussian samples at different blur standard deviations, and applies zero-padding at image edges.Source: §C.6 · p. 29
  • 22.44 FID for convolutional Gaussian blur versus 18.96 for the DCT-based method on CIFAR-10; the authors say DCT may have a minor edge.The convolutional model used otherwise the same parameters as the standard CIFAR-10 model.Source: §C.6 · pp. 29–30
  • Finite-difference heat-equation simulation would require many sequential computations for large blur widths, making it especially slow for larger images.The passage relates long simulation times to t = σB^2/2.Source: §C.6 · p. 30

D SAMPLE VISUALIZATIONS

  • The visualizations cover uncurated samples across datasets, smooth output interpolations, δ behaviour, colour-feature disentanglement, and Euclidean nearest neighbours of generated samples.The appendix notes that δ = 1.25×σ gives good image quality across datasets and resolutions, and that nearest-neighbour examples show generated samples are not merely approximations of training images.Source: D SAMPLE VISUALIZATIONS · p. 30
  • Interpolating the prior states and noise produces smooth output-image interpolations in the model.Prior states uK are interpolated linearly, while noise vectors are interpolated spherically to maintain their magnitude.Source: D SAMPLE VISUALIZATIONS · pp. 30–31

D.1 ADDITIONAL SAMPLES … D.5 NEAREST NEIGHBOURS ON SAMPLED IMAGES

The supplementary sections show additional generated samples and interpolations, examine the effect of sampling noise, illustrate colour–shape disentanglement, and compare samples with training-image neighbours.

D.1 ADDITIONAL SAMPLES

D.2 INTERPOLATIONS

D.3 EFFECT OF SAMPLING NOISE δ

  • Across CIFAR-10, LSUN-Churches, AFHQ, and FFHQ examples, changes are slow below δ = σ = 0.01, and δ = 1.25 × σ is presented as a good default across datasets.The figures fix sampling noise once per column and scale it across δ values for comparison.Source: Figure 32 · p. 45, Figure 33 · p. 46, Figure 34 · p. 47
  • On MNIST, the model performs well across a wide range of δ, including deterministic sampling, while sufficiently high sampling noise degrades results.The passage suggests this robustness may reflect the lack of small-scale detail beyond digit edges; with δ = 0, the model can reshape prior image mass into simple shapes.Source: Figure 37 · p. 50

D.4 DISENTANGLEMENT OF COLOUR AND SHAPE BY FIXING THE SAMPLING NOISE

  • On FFHQ 128×128, changing only the initial state p(uK) while fixing the noise steps produces images with similar characteristics but different average colours; sample quality is somewhat lower than for the regular FFHQ model.Source: Figure 38 · p. 51

D.5 NEAREST NEIGHBOURS ON SAMPLED IMAGES

Related papers

Loading 2206.13397v7…