PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Modelling With Inverse Heat Dissipation

Severi Rissanen, Markus Heinonen, Arno Solin

arXiv:2206.13397v7cs.CVcs.LGstat.ML

TL;DR

Diffusion models do not explicitly account for images’ multi-scale structure. The paper introduces a diffusion-like model that stochastically reverses heat dissipation, yielding smooth interpolation, latent disentanglement, and other emergent properties while connecting its inductive bias to standard diffusion models.

  • Problem

    Diffusion models successfully generate images but do not explicitly consider natural images’ inherent multi-scale structure.

  • Method

    The inverse heat dissipation model interprets a noisy forward heat-equation solution as inference in a diffusion-like latent variable model and stochastically reverses it.

  • Results

    The model exhibits smooth interpolation, latent disentanglement of overall colour and shape, data efficiency, and an explicit multi-scale generative process.

  • Takeaways & Limitations

    Heat dissipation offers a natural image-generative sequence whose inductive bias can clarify coarse-to-fine behaviour in related diffusion models.

  • Takeaways & Limitations

    The model lacks some established diffusion-model guarantees, and its image quality remains below state-of-the-art diffusion models and GANs.

Abstract

from arXiv · show

While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to-fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise as a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.

1 INTRODUCTION

The paper addresses the lack of explicit multi-scale structure in diffusion models by using heat dissipation as a natural coarse-to-fine image process. It proposes stochastically reversing this process to create a diffusion-like generative model.

  • Diffusion models erase image information forward and regenerate images iteratively, but do not explicitly encode natural images’ multi-scale structure.
  • The heat equation produces continuously varying lower-resolution images by averaging pixels and removing fine detail without reducing pixel count.
  • Scale-space provides a principled alternative to pixel-subsampling pyramids and satisfies properties including rotational symmetry, shift invariance, and scale invariance.
  • The proposed inverse heat dissipation model reverses the heat equation so erased information is reconstructed through multiple plausible stochastic paths.
  • The paper reports disentangled colour and shape, smooth interpolation, simpler learned functions, and potential data efficiency as emergent properties of the model.

2 METHODS

IHDM averages images by evolving each colour channel under the heat equation, then learns a stochastic reverse process that reconstructs progressively finer structure. Additive forward noise makes reverse conditionals well-defined, while training alternates noisy deblurring predictions and Gaussian-noise sampling.

  • Forward process: The forward process averages image channels through the heat equation, ultimately projecting each channel toward its mean and the image toward a three-dimensional colour representation.Neumann boundary conditions impose zero derivatives at image boundaries.
  • Forward process: The heat-equation solution decays frequency components at rates determined by the Laplacian eigenvalues, enabling efficient evaluation in a cosine basis.The finite image grid supplies a Nyquist cutoff, and the diagonalized evolution operator is fast to compute.
  • Generative model formulation: A small forward noise level σ converts deterministic heat dissipation into a variational inference process with well-defined stochastic reverse conditionals.The noise acts as an error tolerance, allowing blurred forward paths to merge probabilistically.
  • Generative model formulation: The reverse process is a Gaussian Markov chain from a low-dimensional blurred prior to the observed image, with δ controlling sampling noise and trajectory stochasticity.The prior is estimated with a Gaussian kernel density model because terminal blurred images are effectively low-dimensional.
  • Training and sampling: Training uses noisy heat-evolved images to predict a slightly less blurred state with mean-squared losses, while sampling alternates neural-network deblurring and Gaussian-noise injection.A skip connection parameterizes the prediction as µθ(uk, k) = uk + fθ(uk, k).
  • Frequency behaviour: Natural-image spectra help explain coarse-to-fine generation: isotropic noise masks higher frequencies first, whereas IHDM explicitly decays high frequencies faster and creates an effective resolution range.Natural-image power spectral density approximately follows 1/f^α, often with α ≈ 2.

3 EXPERIMENTS

Experiments examine IHDM’s coarse-to-fine generation, noise dependence, emergent disentanglement and interpolation, inductive bias, and few-shot behavior, while quantitative quality remains below leading diffusion and GAN results.

  • Generative sequences: Generation starts from a blank image and progressively adds finer-scale structure, with large-scale structure determined earlier in the stochastic sequence.The reverse heat process redistributes image mass while preserving average colour.
  • Quantitative evaluation: 18.96 CIFAR-10 FID for IHDM compares with 3.17 for DDPM and 1.85 for current state-of-the-art methods.The authors describe IHDM image quality as promising but not yet competitive with state-of-the-art diffusion models and GANs.
  • The importance of σ: Non-zero training noise σ regularizes the reverse process; setting σ=0 produces essentially random images even on MNIST.Directly reversing the heat equation is described as unstable because the inverse is extremely ill-conditioned.
  • Noise effects: The optimal sampling-noise ratio δ/σ is approximately 1.25–1.3; δ=0 mainly sharpens the prior, while larger values add detail before producing noise.A ratio of 1.25 is reported as a good default across data sets.
  • Disentanglement: IHDM can disentangle overall colour from image shape by varying the colour-containing prior while fixing the generative noise steps.The experiment uses a 128×128 FFHQ model with σB,max=128.
  • Smooth interpolation: IHDM produces perceptually smooth latent interpolations, whereas DDPM interpolations pass through features absent from both endpoints.The IHDM latent is hierarchical, with different steps corresponding to different resolutions.
  • Inductive bias: IHDM learned functions are circularly symmetric and localized, unlike the more global and complex input dependence observed for DDPM.The comparison uses input-gradient visualizations near the end of generation.
  • Few-shot learning: With only the first 20 MNIST digits, IHDM produces meaningful generalisation, while DDPM either fails to produce convincing samples or overfits.This result is presented as evidence of data efficiency from the generative process’s inductive bias.

4 RELATED WORK

Related work situates IHDM among diffusion, multi-resolution generative, and inverse-problem models. Its distinguishing focus is explicitly reversing heat-equation blur rather than extending standard diffusion with blur or treating deblurring as the primary task.

  • Diffusion models: Diffusion research developed from score-based models into unified reverse-SDE and high-performing diffusion frameworks.The cited progression includes DDPM connections, reverse SDEs, and state-of-the-art ImageNet performance.
  • Resolution hierarchies: Multi-resolution hierarchies have improved generative modelling in GANs, VAEs, and autoregressive models through explicit resolution changes.Examples include stacked upsampling GANs, progressive and StyleGAN architectures, resolution-based VAEs, and multi-scale pixelRNN.
  • Inverse problems: Prior generative models have applied diffusion and related methods to deblurring, super-resolution, and other inverse problems.The paper distinguishes these applications from its generative-model formulation based on reversing heat dissipation.
  • Parallel work: Concurrent work incorporates Gaussian blur into standard diffusion, whereas IHDM explicitly reverses the heat equation and departs from the standard Markovian forward framework.The comparison identifies blur-plus-noise diffusion and inverse-heat generation as distinct approaches.

5 DISCUSSION AND CONCLUSIONS

The paper proposes inverse heat dissipation as a diffusion-like generative model and presents it as an initial framework for studying inductive biases. It reports smooth interpolation, latent disentanglement, and data efficiency, while identifying open questions in the probabilistic formulation and neural architecture.

  • 5 DISCUSSION AND CONCLUSIONS: The inverse heat dissipation model explicitly reverses the heat equation to explore inductive biases in diffusion-like generative models.Its forward process averages images, while stochastic reverse steps reconstruct plausible images.
  • 5 DISCUSSION AND CONCLUSIONS: The approach shows smooth interpolation, latent disentanglement, and data efficiency as useful emergent properties.The paper also reports generalization from only the first 20 MNIST digits.
  • 5 DISCUSSION AND CONCLUSIONS: Future work includes analyzing the probabilistic formulation and improving the neural network, whose architecture may be optimized for standard diffusion models rather than IHDM.The paper also suggests extending coarse-graining operators to domains such as audio and graphs.
  • 5 DISCUSSION AND CONCLUSIONS: Heat dissipation is implemented through a Laplacian-based PDE, eigenbasis projection, and discrete cosine transforms on regular image grids.The chosen Neumann boundary conditions eventually average each channel to its original mean.
  • 5 DISCUSSION AND CONCLUSIONS: The asymptotic analysis connects the model to denoising score matching and Langevin dynamics but is not a derivation of the practical sampling procedure.In practice, the model takes directed steps backward through the heat equation rather than slowly shifting distributions with Langevin dynamics.

A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS

The analysis shows that additive isotropic Gaussian noise raises each frequency component's expected power spectral density by the noise variance. Consequently, components below the noise floor become indistinguishable from noise.

  • A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS: PSD(u)i is defined as the squared magnitude of the projection onto the ith DCT or DFT frequency component.The one-dimensional PSD is obtained by averaging components with equal frequency.
  • A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS: The expected PSD of an image with isotropic noise equals the original image PSD plus the noise variance.This follows for additive noise with covariance γ2I.
  • A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS: Frequency components with lower PSD than the noise variance are effectively drowned out by isotropic noise.Averaging equal-frequency components changes the expression only by replacing individual PSD values with their mean.
  • A.5 ANALYSIS FOR THE POWER SPECTRAL DENSITY IN THE DIFFUSION FORWARD PROCESS: An individual PSD component is distributed as the sum of a normally distributed random variable and a chi-squared variable.The analysis uses the fact that projecting isotropic zero-mean Gaussian noise onto one orthonormal direction yields a one-dimensional Gaussian.

B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS

The implementation uses a U-Net-based architecture and a heat-equation forward process evaluated efficiently in the DCT basis. Its logarithmic schedule reduces effective frequency content at an approximately constant rate.

  • B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS: The model uses a U-Net with self-attention at multiple low-resolution feature maps and parameterizes transitions as µθ(uk, k) = uk+fθ(uk, k).The architecture otherwise follows established diffusion-model designs.
  • B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS: FID evaluation uses 50,000 samples for most datasets and 10,000 for 256×256 datasets, where lower sample counts slightly overestimate scores.Generated and reference images are resized to 299×299 before InceptionV3 evaluation.
  • B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS: The experiments generally use σ = 0.01 and K = 200, with K = 400 reported to slightly improve LSUN Churches sample quality.The study states that σ was not extensively optimized.
  • B.1 NEURAL NETWORK ARCHITECTURE AND HYPERPARAMETERS: The logarithmic σB and tk schedule leaves fewer frequency components at an approximately constant rate, slowing somewhat near the end.Components below the σ2 = 0.01^2 noise level become indistinguishable from forward-process noise.

C.1 ROBUSTNESS TO σ AND δ

The CIFAR-10 robustness study finds that performance depends primarily on the ratio δ/σ rather than either absolute parameter value. Across tested σ values, visual differences are qualitatively small when δ = 1.3 × σ.

  • C.1 ROBUSTNESS TO σ AND δ: The model is fairly robust to σ choices provided σ is not too close to zero.The authors identify δ/σ, rather than absolute δ, as the key parameter.
  • C.1 ROBUSTNESS TO σ AND δ: The optimal FID values occur at approximately the same δ/σ ratio, about 1.3, across tested σ values.The study evaluates σ from 0.005 to 0.02 and varies δ.
  • C.1 ROBUSTNESS TO σ AND δ: Images generated with different σ values and δ = 1.3 × σ show no large qualitative visual differences.The comparison is qualitative rather than a reported numerical image metric.

C.2 DATA EFFICIENCY / FEW-SHOT GENERALIZATION

IHDM generalizes beyond extremely small training sets, while the comparison diffusion model either fails to generate plausible digits or overfits. Its likelihood analysis also reveals that the δ value optimizing NLL differs from the one optimizing FID.

  • C.2 DATA EFFICIENCY / FEW-SHOT GENERALIZATION: With only the first 20 MNIST digits, IHDM generalizes to new digit shapes without noticeably overfitting.The diffusion comparison either fails to produce plausible digits or completely overfits the training data.
  • C.2 DATA EFFICIENCY / FEW-SHOT GENERALIZATION: The model combines learned features across resolution scales to form new images and meaningful variation from very few examples.This provides an explicit multi-resolution mechanism for generalization in low-data regimes.
  • C.3 THE OPTIMAL δ WITH RESPECT TO MARGINAL LOG-LIKELIHOOD VS. FID: The δ value with the lowest NLL does not match the δ value with the lowest FID.The lowest NLL occurs near σ = 0.01, whereas the lowest FID occurs near 0.01325, the chosen δ.
  • C.3 THE OPTIMAL δ WITH RESPECT TO MARGINAL LOG-LIKELIHOOD VS. FID: Without the MSE term in the KL divergences, differentiation gives an optimal δ equal to σ.Including the MSE term shifts the optimum toward a higher δ, although the shift is small when the MSE loss is low.
  • C.3 THE OPTIMAL δ WITH RESPECT TO MARGINAL LOG-LIKELIHOOD VS. FID: The model appears to operate in a regime where σ^2/δ terms dominate NLL scores, complicating simultaneous optimization of FID and marginal likelihood.The authors suggest studying the interaction of these terms to improve marginal log-likelihood maximization.

C.4 EVALUATION OF THE PRIOR IN TERMS OF OVERLAP WITH THE TEST SET

The overlap between blurred training and test distributions provides a prerequisite measure for expected generalization because the blurred training distribution supplies the prior.

  • C.4 EVALUATION OF THE PRIOR IN TERMS OF OVERLAP WITH THE TEST SET: LK measures overlap between blurred training and test distributions and can indicate whether the model may generalize beyond the training set.The prior p(uK) is sampled from the training data, making this overlap relevant to generalization.
  • C.4 EVALUATION OF THE PRIOR IN TERMS OF OVERLAP WITH THE TEST SET: When σB,max ≈ 0, the blurred distributions barely overlap and the generative process approaches memorization of the training set.The passage contrasts this with stronger averaging at the end of the forward process.

C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ

Non-zero training noise is necessary for sensible IHDM behavior: without it, reverse dynamics amplify errors into random patterns, whereas noise regularizes the learned process.

  • C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ: Training noise σ is necessary for the model to be defined in a sensible way.The σ = 0 model produces essentially random patterns from MNIST.
  • C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ: With σ = 0, the network learns an unstable sharpening filter whose small reverse-step errors are amplified.The generated image quickly blows up from the flat prior into a random pattern.
  • C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ: A non-zero σ makes the model more robust and forces the neural network to account for the training-data distribution.This contrasts with directly solving the extremely ill-conditioned inverse heat equation.
  • C.5 THE IMPORTANCE OF NON-ZERO TRAINING NOISE σ: The few-shot experiment uses the first 20 MNIST digits to test whether the model avoids overfitting in a severe low-data regime.The corresponding figure compares training coverage, evaluation loss, and generated samples for diffusion and IHDM.

C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER

A Gaussian-blur convolutional implementation is viable but somewhat slower and less directly interpretable than the DCT-based heat process. On CIFAR-10, it achieves a slightly worse FID.

  • C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER: The convolutional blur implementation is computationally slower and does not directly expose frequency-decay or heat-equation boundary-condition intuitions.The DCT-based approach makes these properties explicit.
  • C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER: The convolutional model uses a (2N − 1)×(2N − 1) Gaussian kernel so sufficiently large blur widths let every pixel affect every other pixel.Zero-padding is used at the image edges.
  • C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER: 22.44 versus 18.96 FID: the convolutional filter performs slightly worse than the DCT-based method on CIFAR-10.The authors describe the difference as minor.
  • C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER: Finite-difference simulation should work in principle but becomes slow for large blur widths because it requires many sequential computations.The practical cost is especially relevant for larger images.
  • C.6 COMPARISON WITH GAUSSIAN BLUR IMPLEMENTED WITH A CONVOLUTIONAL FILTER: The model’s interpolation procedure linearly interpolates prior states and uses spherical interpolation for Gaussian noise vectors.Spherical interpolation preserves movement along a hypersphere between approximately orthogonal, equal-magnitude noise vectors.

D.1 ADDITIONAL SAMPLES

The appendix presents uncurated samples across MNIST, CIFAR-10, LSUN-CHURCHES, AFHQ, and FFHQ, alongside interpolations, shared-initial-state comparisons, and sampling-noise studies. It also illustrates colour–shape disentanglement and nearest-neighbour comparisons.

  • Uncurated samples are shown for MNIST and CIFAR-10, including CIFAR-10 samples with FID 18.96.
  • Interpolations between two random images are shown for LSUN-CHURCHES 128×128, FFHQ 256×256, and AFHQ 64×64.
  • Varying the sampling-noise parameter δ is illustrated across datasets, with δ = 1.25 × σ identified as a good default across datasets and resolutions.The figures highlight δ = σ = 0.01 and describe changes before that value as slow.
  • The MNIST model performs well across a wide range of δ values, including deterministic sampling.
  • Changing only the initial state produces FFHQ images with similar characteristics but different average colours, while nearest-neighbour comparisons are shown for FFHQ, AFHQ, and LSUN-CHURCHES.The colour–shape example uses FFHQ 128×128 and notes somewhat lower sample quality than the regular FFHQ model.
Loading 2206.13397v7…