PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Modeling by Estimating Gradients of the Data Distribution

Yang Song, Stefano Ermon

arXiv:1907.05600v3cs.LGstat.ML

TL;DR

Score-based generative modeling struggles when data lie on low-dimensional manifolds and naive Langevin sampling mixes poorly. This paper addresses these issues with noise-conditioned score estimation and annealed Langevin dynamics, producing high-quality images and a state-of-the-art CIFAR-10 inception score of 8.87.

  • Problem

    Score-based generative modeling has undefined or inconsistent scores on low-dimensional data manifolds, while existing generative models face architectural, training-stability, or evaluation limitations.

  • Method

    The method jointly estimates scores of Gaussian-noise-perturbed data with a noise-conditioned network and samples using Langevin dynamics while gradually reducing the noise level.

  • Results

    8.87 CIFAR-10 inception score, with samples comparable to modern likelihood-based models and GANs across MNIST, CelebA, and CIFAR-10.

  • Takeaways & Limitations

    The framework generates high-quality images without adversarial training, MCMC sampling during training, or specialized model architectures.

  • Takeaways & Limitations

    The sampling analysis assumes finite-step Langevin error is negligible when the step size is small and the number of steps is large.

Abstract

from arXiv · show

We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.

1 Introduction

The paper introduces generative modeling by estimating the score of the data distribution with score matching and generating samples through Langevin dynamics. Gaussian perturbations at multiple noise levels address manifold-supported data and low-density regions, enabling a single noise-conditioned score network and tractable, non-adversarial training.

  • 1 Introduction: The paper proposes learning the gradient of the log data density with score matching and generating samples by moving random initial points toward high-density regions using Langevin dynamics.The score is a vector field pointing where the log data density increases most.
  • 1 Introduction: Low-dimensional data manifolds can make ambient-space scores undefined, while scarce training data in low-density regions can impair score estimation.These are identified as the two main challenges for direct score-based generation.
  • 1 Introduction: The method perturbs data with Gaussian noise at multiple magnitudes, jointly estimating all corresponding scores with one network conditioned on the noise level.Noise prevents the perturbed distribution from collapsing onto a low-dimensional manifold and improves estimation in low-density regions.
  • 1 Introduction: The training objective is tractable for almost all score-network parameterizations and requires no special constraints, adversarial training, MCMC sampling, or other approximations during training.The objective also supports quantitative comparison of models on the same dataset.
  • 1 Introduction: The approach is experimentally evaluated on MNIST, CelebA, and CIFAR-10 to demonstrate its efficacy.These datasets are presented as the experimental benchmarks for the proposed framework.

2 Score-based generative modeling

Score-based generative modeling learns data-distribution gradients with score matching and uses them in Langevin dynamics to generate samples. Denoising score matching improves scalability by estimating scores of perturbed data distributions.

  • Score-based generative modeling: The framework has two ingredients: score matching estimates ∇x log pdata(x), while Langevin dynamics uses the learned score to generate samples.The score network sθ is trained to approximate the data-distribution score without first estimating pdata(x).
  • Denoising score matching: Denoising score matching perturbs data with prescribed noise and completely avoids computing tr(∇xsθ(x)), addressing score matching’s poor scalability to deep, high-dimensional models.It estimates the score of the perturbed distribution qσ rather than directly estimating the original data distribution’s score.
  • Denoising score matching: The optimal denoising score equals ∇x log qσ(x), which approximates ∇x log pdata(x) only when the perturbation noise is sufficiently small.This approximation depends on qσ(x) being close to pdata(x).
  • Sliced score matching: Sliced score matching estimates scores for the original distribution using random projections, but requires around four times more computation than denoising score matching.Forward-mode automatic differentiation efficiently computes the projected Jacobian term.
  • Langevin dynamics: Langevin dynamics recursively updates samples using the score and Gaussian noise, becoming exact as ϵ → 0 and T → ∞ under regularity conditions.For small ϵ and large T, the work assumes discretization error is negligible and typically ignores Metropolis-Hastings correction.

3 Challenges of score-based generative modeling

Naïve score-based generative modeling faces two major obstacles: scores may be undefined on low-dimensional manifolds, and sparse low-density regions impair score estimation and Langevin sampling. Separated modes can therefore receive incorrect relative weights, even when Langevin dynamics use exact scores.

  • Manifold hypothesis: Under the manifold hypothesis, the ambient-space score ∇x log pdata(x) is undefined when data are confined to a low-dimensional manifold.Real-world data often concentrate on low-dimensional manifolds embedded in high-dimensional ambient spaces.
  • Manifold hypothesis: Perturbing CIFAR-10 data with Gaussian noise improves the otherwise irregular sliced score matching behavior observed without noise.The comparison trains a ResNet with sliced score matching on original versus perturbed CIFAR-10 images.
  • Low-density regions: Score matching estimates can be inaccurate in low-density regions because those regions contain too few data samples.A toy mixture-of-Gaussians experiment finds reliable score estimation only near high-density modes.
  • Low-density regions: When modes are separated by low-density regions, Langevin dynamics may fail to recover their relative weights and may not converge to the true distribution.For disjoint mixture components, the score within each support does not depend on the mixture weight π, so Langevin samples do not depend on π.
  • Low-density regions: Langevin dynamics with exact scores produce incorrect relative density between mixture modes, whereas annealed Langevin dynamics recover the relative weights faithfully.This comparison is reported for the same Gaussian mixture used in the toy analysis.

4 Noise Conditional Score Networks: learning and inference

The section introduces Noise Conditional Score Networks, which jointly estimate scores for Gaussian-perturbed data at multiple noise levels. It pairs this training objective with annealed Langevin dynamics, progressively reducing noise during sampling to move from easy-to-sample perturbed distributions toward the data distribution.

  • Motivation: Gaussian perturbation makes score estimation well-defined by removing low-dimensional-manifold support and improves Langevin mixing by filling low-density regions and reducing mode isolation.These effects make sampling from the largest-noise perturbed distribution more reliable and provide a basis for subsequent annealing.
  • Model: A single Noise Conditional Score Network jointly estimates ∇x log qσ(x) for all selected noise levels, with σ1 large enough to mitigate manifold difficulties and σL small enough to limit data distortion.The network is conditioned on σ and outputs a D-dimensional score when x ∈ R^D.
  • Learning: Denoising score matching trains the network efficiently for Gaussian-perturbed distributions, and with sufficient capacity its optimum equals the true score at every noise level.Sliced score matching can also train NCSNs empirically, but denoising score matching is slightly faster and naturally fits the task.
  • Learning: The objective requires no adversarial training, surrogate losses, or score-network sampling during training, and can quantitatively compare NCSNs when λ(·) and the noise levels are fixed.It also does not require special architectures for tractability.
  • Inference: Annealed Langevin dynamics initializes from a fixed prior, samples successively from qσ1(x) through qσL(x), and uses each stage’s final samples to initialize the next lower-noise stage.When σL ≈ 0, the final perturbed distribution is close to pdata(x); adjacent distributions differ only slightly, supporting progressive refinement.
  • Inference: L = 10, σ1 = 10 and σ10 = 0.1; annealed Langevin dynamics correctly recovered the relative weights of two well-separated Gaussian modes, whereas standard Langevin dynamics failed.The noise levels were chosen as a geometric progression in the toy experiment.

5 Experiments

Experiments show that NCSNs generate high-quality images on MNIST, CelebA, and CIFAR-10, with an unconditional CIFAR-10 inception score of 8.87. The models also learn representations that support diverse image inpainting for arbitrary-shaped occlusions.

  • Setup: The experiments use MNIST, CelebA, and CIFAR-10, with L = 10 noise levels geometrically spanning σ1 = 1 to σ10 = 0.01.Annealed Langevin dynamics uses T = 100, ϵ = 2 × 10^-5, and uniform-noise initial samples.
  • Image generation: NCSNs generate high- or comparable-quality images on MNIST, CelebA, and CIFAR-10 relative to modern likelihood-based models and GANs.Samples evolve from pure random noise to high-quality images through annealed Langevin dynamics.
  • Quantitative evaluation: 8.87 is the unconditional CIFAR-10 inception score, exceeding most reported values for class-conditional generative models.The CIFAR-10 FID score is 25.32, comparable to top existing models such as SNGAN.
  • Image inpainting: Score networks learn generalizable, semantically meaningful representations that produce diverse image inpaintings.A simple modification of annealed Langevin dynamics handles occlusions of arbitrary shapes, unlike PixelCNN's raster-scan-order limitation.

6 Related work

The approach relates to Markov-chain generative methods and score-matching energy-based models, but differs by training a Langevin score function without simulating chains during training. Unlike likelihood-based transition-operator methods, it uses score matching as its objective.

  • Markov-chain generative methods: The method trains the score function used in Langevin dynamics, paralleling generative stochastic networks that train Markov chains whose equilibrium distribution matches the data distribution.GSNs use denoising autoencoders, whereas this approach directly trains the score function for sampling.
  • Likelihood-based methods: Infusion Training, Variational Walkback, and NET use multiple noise levels or temperatures while maximizing evidence lower bounds for suitable marginal likelihoods.These methods tend to produce blurry image samples, similar to variational autoencoders.
  • Training objective: Unlike likelihood-based transition-operator methods, the approach uses score matching rather than likelihood as its training objective.The passage contrasts its objective with methods such as Infusion Training and Variational Walkback.
  • Training procedure: Unlike GSNs, NET, Infusion Training, and Variational Walkback, the approach does not sample from a Markov chain during training, improving training efficiency and scalability.The compared methods simulate chains to generate negative samples or compute training losses.
  • Energy-based models: Recent energy-based approaches have used contrastive divergence and Langevin dynamics for both training and testing, unlike this method.Score-matching methods have also been described as non-scalable or unable to produce samples comparable to variational autoencoders or GANs.

7 Conclusion

The paper proposes score-based generative modeling, combining score matching to estimate data-density gradients with Langevin dynamics for sampling. It addresses naïve-method challenges using Noise Conditional Score Networks and annealed Langevin dynamics without adversarial training, training-time MCMC, or special architectures.

  • 7 Conclusion: The framework estimates gradients of data densities via score matching and generates samples via Langevin dynamics.This defines the paper’s central score-based generative modeling approach.
  • 7 Conclusion: Noise Conditional Score Networks and annealed Langevin dynamics address challenges arising from a naïve application of score-based sampling.The networks are trained for score estimation, while annealed Langevin dynamics is used for sampling.
  • 7 Conclusion: The approach requires no adversarial training, no MCMC sampling during training, and no special model architectures.These are stated requirements of the proposed framework.

A Architectures … B Additional experimental details

The NCSN architecture combines conditional normalization, dilated convolutions, and a RefineNet-based U-Net design. Conditional normalization incorporates noise-level information while addressing color shifts caused by standard instance normalization.

  • A Architectures: NCSNs use instance normalization, dilated convolutions, and U-Net-type architectures as their three main architectural components.The score networks are implemented in PyTorch, with code and checkpoints publicly available.
  • A.1 Instance normalization: Conditional instance normalization lets sθ(x, σ) use different learned scales and biases for each noise level σ_i.This conditions score prediction on σ through parameters indexed by the noise-level set {σ_i}^L.
  • A.1 Instance normalization: Standard instance normalization removes feature-map means, which can cause shifted colors in generated images.The limitation arises because the means µ_k are discarded for different feature maps.
  • A.1 Instance normalization: CondInstanceNorm++ restores mean information by computing statistics of feature-map means and adding a learned parameter α.The modification uses the mean and standard deviation of the µ_k values, denoted m and v.
  • A.1 Instance normalization: The architecture places CondInstanceNorm++ before every convolutional and pooling layer.This modification is used throughout the NCSN architecture.
  • A.2 Dilated convolutions: Dilated convolutions enlarge receptive fields while preserving feature-map resolution and location information, replacing all subsampling layers except the first.Their use is motivated by preserving spatial information at larger feature-map resolutions.
  • A.3 U-Net architecture: The score network uses RefineNet, a modern U-Net variant that combines skip connections with ResNet designs.Skip connections transfer lower-level information from shallow layers to deeper layers, including location and shape information.
  • A.3 U-Net architecture: Experiments use a 4-cascaded RefineNet with pre-activation residual blocks, no batch normalization, and average pooling replacing max pooling.Batch normalization is replaced by CondInstanceNorm++, which is also added before convolutions and average-pooling layers in Refine Blocks.

B.1 Toy experiments … C Samples

The experiments evaluate score-based sampling on toy mixtures, image generation on MNIST, CelebA, and CIFAR-10, and image inpainting with annealed Langevin dynamics. The supplied passages specify experimental settings, baselines, evaluation procedures, and the inpainting algorithm.

  • B.1 Toy experiments: Toy experiments use a 3-layer MLP with 128 hidden units, softplus activations, sliced score matching, 10000 Adam iterations, learning rate 0.001, and batch size 128.The experiment used an Intel Core i7 GPU with 2.7GHz.
  • B.1 Toy experiments: Toy sampling generates 1280 samples per Fig. 3 subfigure from uniform initializations in [−8, 8] × [−8, 8].Langevin dynamics uses T = 1000 and ϵ = 0.1, while annealed Langevin dynamics uses T = 100, L = 10, and ϵ = 0.1.
  • B.2 Image generation: Image-generation models randomly flip CelebA and CIFAR-10 images and train with Adam for 200000 iterations at learning rate 0.001 and batch size 128.For CIFAR-10 and CelebA, checkpoints are selected using the smallest FID score among 1000 generated images per checkpoint.
  • B.2 Image generation: Reported MNIST, CelebA, and CIFAR-10 inception and FID scores are computed with official OpenAI and TTUR code using 50000 samples.MNIST uses one Titan XP GPU, while CelebA and CIFAR-10 use two Titan XP GPUs.
  • B.2 Image generation: The image-generation baseline uses the same score network conditioned only on one noise level, σ1 = 0.01, and samples with Langevin dynamics using ϵ = 2 × 10−5 and T = 1000.This isolates the effect of conditioning the score network on multiple noise levels.
  • B.3 Image inpainting: Image inpainting uses Algorithm 2, annealed Langevin dynamics, and the same hyperparameters as annealed Langevin dynamics for image generation.The algorithm takes a mask m and given image x, iterates over noise levels, and reapplies the observed regions during sampling.

C.1 Samples from the baseline models · C.2 Nearest neighbors

The baseline model produces uncurated samples on MNIST, CelebA, and CIFAR-10, with intermediate Langevin states illustrating the sampling trajectory. Nearest neighbors are evaluated in both image space and pretrained Inception V3 feature space.

  • C.1 Samples from the baseline models: Uncurated baseline-model samples are shown for MNIST, CelebA, and CIFAR-10.The samples are presented without curation across all three datasets.
  • C.1 Samples from the baseline models: Figure 7 presents samples generated by the baseline model across three image datasets.The datasets are MNIST, CelebA, and CIFAR-10.
  • C.1 Samples from the baseline models: Intermediate samples from Langevin dynamics are visualized for the baseline model.The figure displays intermediate states produced during Langevin dynamics.
  • C.2 Nearest neighbors: Nearest neighbors are measured by the ℓ2 distance between images.NCSN samples appear left of the red vertical line, while training-set nearest neighbors appear on the right.
  • C.2 Nearest neighbors: Image-space nearest-neighbor comparisons place NCSN samples beside their training-dataset neighbors.The comparison uses ℓ2 distance between images and separates the two groups with a red vertical line.
  • C.2 Nearest neighbors: Nearest neighbors are also measured by ℓ2 distance in pretrained ImageNet Inception V3 feature space.NCSN samples are shown left of the red vertical line and training-dataset nearest neighbors on the right.

C.3 Extended samples · C.4 Extended intermediate samples from annealed Langevin dynamics · C.5 Extended image inpainting results

The paper provides extended visual results for samples, annealed Langevin dynamics intermediates, and image inpainting across MNIST, CelebA, and CIFAR-10. The inpainting figures compare occluded inputs with original images.

  • C.3 Extended samples: Extended MNIST samples are shown.
  • C.3 Extended samples: Extended CelebA samples are shown.
  • C.3 Extended samples: Extended CIFAR-10 samples are shown.
  • C.4 Extended intermediate samples from annealed Langevin dynamics: Extended intermediate CelebA samples from annealed Langevin dynamics are shown.
  • C.4 Extended intermediate samples from annealed Langevin dynamics: Additional extended intermediate CelebA samples from annealed Langevin dynamics are shown.
  • C.5 Extended image inpainting results: Extended CelebA image inpainting results compare occluded images with original images.
  • C.5 Extended image inpainting results: Extended CIFAR-10 image inpainting results compare occluded images with original images.
Loading 1907.05600v3…