PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Energy-Based Models by Diffusion Recovery Likelihood

Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, Diederik P. Kingma

arXiv:2012.08125v2cs.LGstat.ML

TL;DR

High-dimensional EBMs are difficult to train and sample because marginal-distribution MCMC is challenging. The paper learns a sequence of noisy EBMs with diffusion recovery likelihood, using easier conditional sampling to generate images from white noise. It reports high-fidelity generation, including CIFAR-10 FID 9.58 and inception score 8.30, plus realistic long-run chains that support normalized-density estimation.

  • Problem

    Training and sampling EBMs on high-dimensional datasets remains challenging because marginal-distribution MCMC can converge slowly, struggle across modes, and yield malformed energy landscapes.

  • Method

    The method learns a sequence of EBMs on increasingly noisy data with recovery likelihood, then samples conditional distributions progressively from Gaussian white noise.

  • Results

    The method produces high-fidelity image samples, achieving FID 9.58 and inception score 8.30 on CIFAR-10, while long-run conditional MCMC samples remain realistic.

  • Takeaways & Limitations

    Faithful long-run conditional samples enable estimation of the zero-noise partition function and evaluation of normalized data density under the EBM.

  • Takeaways & Limitations

    The partition-function estimate is a stochastic lower bound that converges to the true value as the number of samples grows.

Abstract

from arXiv · show

While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion probabilistic models, we present a diffusion recovery likelihood method to tractably learn and sample from a sequence of EBMs trained on increasingly noisy versions of a dataset. Each EBM is trained with recovery likelihood, which maximizes the conditional probability of the data at a certain noise level given their noisy versions at a higher noise level. Optimizing recovery likelihood is more tractable than marginal likelihood, as sampling from the conditional distributions is much easier than sampling from the marginal distributions. After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution and progressively samples the conditional distributions at decreasingly lower noise levels. Our method generates high fidelity samples on various image datasets. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs. Moreover, we demonstrate that unlike previous work on EBMs, our long-run MCMC samples from the conditional distributions do not diverge and still represent realistic images, allowing us to accurately estimate the normalized density of data even for high-dimensional datasets. Our implementation is available at https://github.com/ruiqigao/recovery_likelihood.

1 INTRODUCTION

EBMs offer desirable probabilistic modeling properties but remain difficult to train and sample on high-dimensional data. Diffusion recovery likelihood addresses this by learning a sequence of noisy EBMs whose conditional distributions support tractable sampling and high-fidelity generation.

  • Motivation: High-dimensional EBM training remains challenging because maximum likelihood requires difficult MCMC sampling from model distributions.Non-convergent chains can produce malformed energy landscapes even when generated samples appear reasonable.
  • Method: Diffusion recovery likelihood learns a sequence of EBMs for increasingly noisy marginal distributions using conditional recovery likelihoods.The conditional distributions invert the diffusion process and are easier to sample from than marginal distributions.
  • Method: The learned model generates images by starting from Gaussian white noise and progressively sampling conditional distributions at decreasing noise levels.Unlike approaches that approximate reverse transitions with normal distributions, the conditional distributions here are derived from marginal EBMs.
  • Results: Long-run MCMC chains from conditional distributions remain realistic, enabling accurate estimation of the zero-noise marginal partition function and normalized EBM density.This addresses a central difficulty of explicit EBMs: obtaining faithful samples for density evaluation.

2 BACKGROUND

An EBM represents a probability density through an energy function and partition function, but high-dimensional normalization and sampling make maximum-likelihood training difficult. MCMC methods such as Langevin dynamics can fail to converge or traverse modes reliably.

  • Energy-based models: An EBM models a data density with an energy function normalized by a partition function.For high-dimensional inputs, the partition function is analytically intractable; image energies are parameterized by a convolutional neural network with scalar output.
  • Maximum likelihood: Maximum-likelihood training compares observed data with synthesized samples drawn from the model distribution.The synthesized expectation is typically approximated using MCMC samples.
  • MCMC sampling: Langevin dynamics updates samples using the energy gradient, a step size, and Gaussian noise.The update is an MCMC procedure used to generate samples from the model distribution.
  • MCMC sampling: High-dimensional and multimodal distributions can make MCMC slow to converge and unable to traverse modes effectively.Training with non-convergent samples can produce malformed energy landscapes, even when model samples look reasonable.

3 RECOVERY LIKELIHOOD

Recovery likelihood replaces difficult marginal EBM sampling with conditional distributions that are localized around noisy observations and easier to sample. The method extends this idea across a diffusion sequence, enabling generation from white noise while connecting to score-based diffusion methods.

  • From Marginal to Conditional: The quadratic term localizes the energy landscape around the noisy observation, reducing multimodality and the burden of MCMC sampling.For small σ, the conditional distribution is approximately a single-mode Gaussian.
  • From Marginal to Conditional: Recovery likelihood trains EBMs using conditional distributions of clean samples given Gaussian-perturbed observations, rather than sampling directly from the marginal model.The conditional energy adds a quadratic localization term around the noisy observation.
  • Maximizing Recovery Likelihood: Under exact synthesis, sufficient data, and a rich enough model, maximizing recovery likelihood yields an unbiased parameter estimate matching pdata(x) and pθ(x).The claim follows the classical maximum-likelihood analysis and is stated under explicit assumptions.
  • Connection to Variational Inference and Score Matching: For small σ, the conditional Gaussian approximation produces a Langevin-like sampling step and aligns recovery-likelihood training with diffusion probabilistic models and denoising score matching.The alignment is approximate and relies on the small-noise regime.
  • Connection to Variational Inference and Score Matching: The diffusion recovery framework permits more flexible choices of time steps and noise magnitude than the normal approximation, which is accurate only for small σ.The normal approximation therefore requires many diffusion steps to work well.
  • Diffusion Recovery Likelihood: The method learns recovery likelihoods across gradually perturbed observations, then samples backward from Gaussian white noise through decreasing noise levels.Each synthesized sample initializes MCMC sampling for the previous diffusion time step.

4 EXPERIMENTS

Experiments evaluate diffusion recovery likelihood for image generation, interpolation, inpainting, ablations, and long-run sampling. The method produces high-fidelity samples, benefits from more diffusion and sampling steps, and supports realistic long-run chains with approximate density estimation.

  • Experimental settings: The experiments use progressive sampling with diffusion time-step settings T6 and T1k, including Langevin dynamics or normal-approximation sampling.T6 uses K = 30 Langevin steps per time step, while T1k samples from the normal approximation.
  • Image generation: The model generates high-fidelity samples across CIFAR-10, CelebA, and LSUN, with CIFAR-10 FID 9.58 and inception score 8.30.The CIFAR-10 result outperforms explicit EBM methods and is superior to a majority of GAN-based methods; CelebA results are comparable to state-of-the-art GAN methods.
  • Image manipulation: The learned model supports smooth interpolation between generated samples and realistic, semantically meaningful inpainting on LSUN church and CelebA images.Inpainting progressively updates masked pixels while keeping observed pixels fixed as noise levels decrease.
  • Ablation study: The diffusion recovery likelihood method outperforms the T = 1, K = 180 marginal-likelihood baseline and reduces per-iteration sampling through diffusion time steps.The baseline uses one time step, making recovery likelihood equivalent to marginal likelihood under the stated setup.
  • Ablation study: Training becomes unstable with too few sampling steps, while increasing K improves sample quality; K = 50 offers no significant gain over K = 30 at higher cost.The authors therefore use K = 30 for image generation on all datasets.
  • Long-run chain analysis: With a thousand diffusion steps, long-run HMC chains remain realistic for 100k total sampling steps, yielding FID 24.89 versus 25.12 with 1k steps.The method also uses these long-run conditional samples to estimate marginal partition-function ratios and the partition function.
  • Long-run chain analysis: The estimated partition function is a stochastic lower bound that converges to the true value as the number of samples grows.The reported test bits per dimension should therefore be interpreted cautiously.

5 CONCLUSION

The paper proposes diffusion recovery likelihood for learning EBMs through diffusion processes. It reports high-quality synthesis and faithful long-run MCMC samples with a thousand noise levels, while identifying higher-resolution and other modalities as future directions.

  • The paper proposes diffusion recovery likelihood, a variant of maximum likelihood estimation applied to diffusion processes, for learning EBMs.
  • The method achieves high-quality image synthesis and faithful long-run MCMC samples when using a thousand noise levels.
  • Future work includes scaling the method to higher-resolution images and investigating other data modalities.

A.1 DERIVATION OF EQUATION 5

The appendix develops recovery-likelihood theory, showing its estimator retains asymptotic normality despite information loss and connects approximations to Langevin sampling and partition-function estimation.

  • Asymptotic behavior: Recovery likelihood yields an unbiased, asymptotically normal estimator of the true parameters, paralleling maximum likelihood estimation.The result assumes a correctly specified model and compares recovery-likelihood information with the Fisher information of MLE.
  • Asymptotic behavior: The recovery-likelihood estimator loses information relative to MLE but remains statistically consistent in the stated asymptotic sense.The appendix explicitly characterizes the loss through a different information matrix while preserving unbiasedness and asymptotic normality.
  • Noisy-score approximation: For small σ, the score of the noisy distribution differs from the clean-data score by O(σ2), making the difference negligible.The score relation is derived through the heat equation and a small-noise expansion.
  • Langevin approximation: The normal-approximation objective has approximately the same learning gradient as original recovery likelihood after one Langevin step.The comparison is made by contrasting the gradients for observed data and samples generated from the conditional distribution.
  • Partition-function estimation: Partition-function ratios between consecutive noise levels can be estimated by importance sampling along the reverse progressive-sampling path.The path starts from a Gaussian terminal distribution and proceeds toward the clean-data distribution.
  • Partition-function estimation: The logarithm of the ratio estimator is generally a stochastic lower bound, but it converges to the unbiased log-ratio as sample count increases.The convergence follows because the estimator variance vanishes as M grows, matching the observed upward stabilization curve.

B EXPERIMENTAL DETAILS

The experiments use Wide ResNet-based models, standard image datasets, and FID and inception scores, with additional checks of sampling behavior across iteration and chain length.

  • Model architecture: The model architecture is based on Wide ResNet, with the number of residual blocks swept as a hyperparameter across resolutions.Each residual block uses two convolutional layers, with zero initialization and trainable channel-wise output scaling on the second layer.
  • Training: Adam is used for all experiments with learning rate 0.0001, while β1, batch size, and training iterations vary by dataset.A smaller β1 is used for high-resolution images to stabilize training.
  • Datasets and metrics: The evaluation covers CIFAR-10, CelebA, and LSUN church-outdoor and bedroom datasets at their specified resolutions.CIFAR-10 uses 32 × 32 images; CelebA is resized to 64 × 64, while LSUN experiments include 128 × 128 categories.
  • Datasets and metrics: FID and inception scores are computed on 50,000 generated samples using the original evaluation implementations.FID is also tracked on 2,500 samples every 15,000 training iterations.
  • Sampling diagnostics: Samples remain realistic after 100k NUTS sampling steps in total, while FID remains stable.The long-run check uses a No-U-Turn Sampler with the step-size schedule obtained for HMC.
  • Sampling diagnostics: Figure 9 evaluates how FID changes with the number of Langevin steps, complementing the long-run NUTS-chain diagnostic.The supplied caption identifies the plotted quantity but does not state the outcome of the comparison.

C.3 ADDITIONAL INTERPOLATION RESULTS

Additional figures present interpolation sequences between generated samples for CelebA faces and LSUN church and bedroom images.

  • CelebA: Interpolation results are shown between the leftmost and rightmost generated samples on CelebA at 64 × 64 resolution.The figure caption identifies the endpoints and image resolution.
  • LSUN church outdoor: Interpolation results are shown between the leftmost and rightmost generated samples on LSUN church outdoor at 128 × 128 resolution.The accompanying text groups this figure with additional interpolation examples.
  • LSUN bedroom: Interpolation results are shown between the leftmost and rightmost generated samples on LSUN bedroom at 128 × 128 resolution.The accompanying text identifies the bedroom category and its inclusion among the additional examples.

C.4 ADDITIONAL IMAGE INPAINTING RESULTS

Additional image-inpainting examples are provided for CelebA faces and LSUN church-outdoor images.

  • Additional inpainting examples: Figures 14 and 15 show additional image-inpainting examples on CelebA at 64 × 64 and LSUN church outdoor at 128 × 128.The passage identifies the datasets and resolutions but does not report a quantitative comparison.

C.5 ADDITIONAL UNCURATED SAMPLES

This section presents uncurated generated samples from learned models across CIFAR-10, CelebA, and LSUN datasets, alongside inpainting results on CelebA and LSUN church outdoor. Reported FID values are included for several LSUN sample sets.

  • Generated samples: Uncurated samples are shown for CIFAR-10 and CelebA 64 × 64.The samples are from learned models under the T6 setting.
  • Generated samples: Uncurated samples are also shown for LSUN church outdoor and bedroom at 128 × 128 and 64 × 64.The reported FID values are 9.76 for LSUN church outdoor 128 × 128, 11.27 for LSUN bedroom 128 × 128, 7.02 for LSUN church outdoor 64 × 64, and 8.98 for LSUN bedroom 64 × 64.
  • Image inpainting: Image inpainting results on CelebA 64 × 64 compare masked images with inpainted images.The masked images appear above the inpainted images.
  • Image inpainting: Image inpainting results on LSUN church outdoor 128 × 128 compare masked images with inpainted images.The masked images appear above the inpainted images.
Loading 2012.08125v2…