PaperlayerView exact revision on arXiv

Source-linked AI summary

Improved Techniques for Training Score-Based Generative Models

Yang Song, Stefano Ermon

arXiv:2006.09011v2cs.LGcs.CVstat.ML

TL;DR

Score-based generative models were limited to low-resolution images and could exhibit unstable sampling or visual quality. The paper analyzes these failures and introduces techniques for noise-scale selection, score estimation, sampling, and weight averaging. Together, the methods scale generation to 64 × 64 through 256 × 256 images with high-fidelity samples.

  • Problem

    Existing score-based models were limited to low-resolution images, while learned-score Langevin sampling and visual quality could become unstable at higher resolutions.

  • Method

    The paper combines theoretical analyses with data-driven noise-scale selection, an amortized noise-conditional score network, sampling optimization, and exponential moving averages of model parameters.

  • Results

    The techniques scale score-based generation to images from 64 × 64 to 256 × 256 and produce sharp, diverse, high-fidelity samples.

  • Takeaways & Limitations

    The proposed techniques improve training and sampling processes and enable high-fidelity image generation at high resolutions without manual tuning.

  • Takeaways & Limitations

    The method assumes sufficient data and model capacity for the optimal noise-conditional score network to match each perturbed distribution's score.

Abstract

from arXiv · show

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

1 Introduction

Score-based generative models avoid adversarial optimization and can produce realistic samples, but existing methods were limited to low-resolution images. The paper introduces techniques that enable scaling to high-resolution generation with sharp, diverse samples.

  • Score-based models represent distributions through score functions and can learn them without adversarial optimization.The score is a vector field indicating where data likelihood increases most rapidly.
  • Existing methods were limited to 32 × 32 images because noise-scale selection and Langevin sampling become problematic at higher resolutions.The recommended noise settings degrade at higher resolution, while sampling can fail or converge slowly in high dimensions.
  • The proposed techniques use theoretical analysis to improve noise-scale selection, sampling, and model training for high-resolution score-based generation.The approach includes analytically computed noise scales, an amortized score network, sampling optimization, and exponential moving averages of model parameters.
  • 64 × 64 to 256 × 256 resolutions are achieved, with samples described as sharp and diverse across FFHQ, LSUN, and CelebA datasets.Figure 1 lists FFHQ at 256 × 256, LSUN datasets at 96 × 96 and 128 × 128, and CelebA at 64 × 64.

2 Background

Score-based generation estimates score functions for noise-perturbed data distributions and uses annealed Langevin dynamics to sample progressively from coarse to fine representations. The section describes the training objective, sampling procedure, and design choices that affect performance.

  • Langevin dynamics: The score function is the gradient of log density, and an estimated score network can replace it in Langevin dynamics.Langevin dynamics starts from a prior sample and iteratively applies the estimated score with Gaussian noise.
  • Score-based generative modeling: Noise-conditional score networks jointly estimate scores for multiple Gaussian-perturbed data distributions.At the optimum, the network matches the score of each perturbed distribution under sufficient data and model capacity.
  • Annealed Langevin dynamics: Annealed Langevin dynamics sequentially samples from distributions at decreasing noise scales, using each stage to initialize the next.The procedure moves from σ1 to σL, with step size αi = ϵσi^2 and T iterations at each scale.
  • Sampling refinement: An extra denoising step after annealed Langevin dynamics often improves FID scores without changing samples' visual appearance.The update uses the final score estimate to remove unwanted Gaussian noise from the terminal sample.
  • Design choices: Successful NCSN training and inference depend on noise scales, noise conditioning, step size, and sampling steps per scale.The paper provides theoretically motivated configurations intended to improve high-resolution performance without manual tuning.

3 Choosing noise scales

The paper develops theoretically motivated rules for selecting noise scales, addressing unresolved choices that limit score-based models on high-resolution images. These rules balance sample diversity, distributional overlap, computational cost, and scalable noise conditioning.

  • Motivation: Single-noise models cannot produce convincing large-image samples, motivating multiple noise scales that combine easier score estimation at high noise with cleaner samples at low noise.High noise facilitates score estimation but corrupts samples, whereas low noise improves sample fidelity while making scores harder to estimate.
  • 3.1 Initial noise scale: Technique 1 sets the initial noise scale σ1 as large as the maximum Euclidean distance between training data points to facilitate transitions between mixture components.The analysis indicates that transition weights can decay exponentially when σ1 is too small relative to data distances.
  • 3.2 Other noise scales: Technique 2 chooses subsequent noise scales to maintain overlap between adjacent perturbed distributions while balancing sampling cost, recommending an overlap constant C ≈ 0.5.Too many scales increase sampling cost, while the original L = 10 setting can provide negligible overlap in the analyzed high-dimensional regime.
  • 3.3 Incorporating the noise information: For high-resolution images, the original noise-conditioning scheme has memory consumption that grows linearly with the number of noise scales and does not apply to architectures without normalization layers.This creates a scalability problem when large σ1 requires many noise scales.
  • 3.3 Incorporating the noise information: Technique 3 parameterizes the score network as sθ(x, σ) = sθ(x)/σ, enabling one unconditional network to handle many or even continuous noise scales.The method achieves similar training losses and better sample quality than the original conditioning approach in the reported experiments.

4 Configuring annealed Langevin dynamics

The paper configures annealed Langevin dynamics by selecting the sampling budget first and then tuning the step size to improve mixing across noise scales. This replaces manually fixed settings with a theoretically motivated procedure.

  • Motivation: The original procedure leaves the appropriate step size ϵ and number of sampling steps T unclear when the noise-scale set changes.The prior recommendation was ϵ = 2 × 10^-5 and T = 100.
  • Analysis: With geometric noise scales, the analyzed variance quantity is identical across adjacent scales and has no explicit dependence on dimensionality D.The shared geometric ratio produces the same quantity across the noise-scale transitions considered in the analysis.
  • Selecting T and ϵ: Technique 4 chooses T as large as the computing budget allows, then selects ϵ to make the analyzed variance factor as close to 1 as possible.This avoids selecting an unnecessarily large T while targeting better mixing across noise scales.

5 Improving stability with moving average

Exponential moving average stabilizes score-model training outputs, reducing FID volatility and visual color artifacts while generally improving FID scores.

  • EMA stabilizes generated-image quality even when the NCSN loss decreases steadily during training.The instability is especially noticeable for larger-resolution images.
  • EMA updates an independent parameter copy after each optimization step and uses that copy for sampling.The update uses momentum m, typically 0.999.
  • FID trajectories for vanilla NCSN are more volatile than those for NCSN with EMA, whose samples also avoid frequent color shifts.The reported FIDs are computed with the denoising step.
  • EMA effectively stabilizes FIDs, removes artifacts, and gives better FID scores in most cases across many image datasets.The authors recommend applying EMA when sampling.

6 Combining all techniques together

Combining the proposed techniques produces NCSNv2, which improves score-based generation across datasets and enables high-fidelity samples at substantially higher resolutions.

  • Combining all techniques together: NCSNv2 combines Techniques 1–5 to work across many datasets, including resolutions previously out of reach for score-based models.The modified model is named NCSNv2.
  • Quantitative results: NCSNv2 has better average FIDs and substantially lower training-time variance than NCSN on CIFAR-10 and CelebA.FID scores are evaluated every 5000 training iterations on 1000 samples.
  • Ablation studies: Removing technique groups from NCSNv2 decreases sample quality visually, while combining all techniques produces the best samples.The ablation groups isolate EMA, noise-scale and sampling-related techniques, and other techniques.
  • Towards higher resolution: Original NCSN produces strong color artifacts at 64 × 64 and fails to preserve correct structure or color beyond 96 × 96.The comparison is made on CelebA and LSUN examples without denoising.
  • Towards higher resolution: NCSNv2 generates high-fidelity samples from 96 × 96 to 256 × 256 and supports smooth interpolations between samples.Noise scales and annealed Langevin step sizes are calculated without manual hyper-parameter tuning.

7 Conclusion

The proposed techniques improve score-based model training and sampling, yielding better sample quality and high-fidelity generation at high resolutions.

  • The techniques improve training and sampling processes, lead to better sample quality, and enable high-fidelity image generation at high resolutions.They work well without manual tuning, although fine-tuning hyper-parameters may improve performance further.

Broader Impact

The paper discusses broad applications and societal consequences of score-based generative models, including both beneficial uses and risks from realistic synthetic images.

  • Potential applications: Score models may support data synthesis, anomaly detection, adversarial-example detection, semi-supervised learning, and reinforcement learning.The authors also suggest similar techniques could apply to speech and behavioral data.
  • Risks and benefits: Image models trained on biased datasets can generate synthetic images that are difficult for humans to distinguish from real images.Such images have already been used to deceive people maliciously.
  • Risks and benefits: The technologies may have positive uses in the arts and as tools for engineering design, while their societal effects depend on application.

A Proofs

The proofs derive radial and distributional properties for Gaussian variables and use asymptotic arguments to establish the stated results.

  • Gaussian radial properties: The radial density is expressed using the chi-squared density, yielding ps(s) = 1/(2^(D/2)Γ(D/2)) s^(D/2−1)e^(−s/2).The derivation starts from the chi-squared density and substitutes r = σ√s.
  • Gaussian radial properties: The proof applies the change-of-variables formula to relate the Gaussian norm r = ∥x∥2 to a chi-squared variable s.For x ∼ N(0, σ2I), the squared normalized norm follows a chi-squared distribution.
  • Asymptotic analysis: For independent Gaussian coordinates, the law of large numbers and central limit theorem provide the relevant high-dimensional limits as D →∞.The coordinates satisfy E[x] = σ2 and Var[x] = 2σ4 in the supplied derivation.

B.1 Network architectures and hyperparameters

The appendix specifies the NCSN/NCSNv2 architectures, optimization settings, datasets, evaluation procedures, and interpolation method used in the experiments.

  • Network architectures: NCSN modifies RefineNet with CondInstanceNorm++, average pooling, and dilated convolutions in the ResNet backend.These changes define the original NCSN architecture used as the baseline.
  • Network architectures: Higher-resolution models use more layers and filters to provide sufficient receptive fields and model capacity.Architectural configurations for NCSN and NCSNv2 are summarized in Tables 2 and 3.
  • Hyperparameters: NCSNv2 hyperparameters configure noise levels and annealed Langevin dynamics according to Techniques 1–4.The table identifies σ1 and L as noise-level parameters and T and ϵ as sampling parameters.
  • Datasets and evaluation: Experiments use CIFAR-10, CelebA, LSUN church_outdoor, bedroom and tower, and FFHQ at resolutions from 32 × 32 to 256 × 256.The datasets and preprocessing procedures are described in the experimental settings.
  • Datasets and evaluation: FID and HYPE∞ evaluate samples, with checkpoints selected by the smallest FID measured every 5000 training iterations on 1000 samples.FID conventions differ across CIFAR-10 and CelebA, while HYPE∞ is computed using the official website.
  • Interpolation: Interpolation between samples is performed by interpolating the Gaussian noise injected during annealed Langevin dynamics.The construction uses the collection of noises across all noise levels and Langevin iterations.

C.1 Additional results without the denoising step

Without denoising, EMA stabilizes score-based training and removes sample artifacts, while FID can disagree with human judgments of visual quality.

  • EMA effects: EMA stabilizes training and removes sample artifacts regardless of whether denoising is used.This effect is demonstrated using FID curves computed without the denoising step.
  • EMA effects: NCSN FIDs show much higher volatility than NCSN with EMA, and vanilla samples often exhibit obvious color shifts.The comparison is reported for FIDs and color artifacts over training.
  • Evaluation metrics: 37.3 on CelebA 64 × 64 is the HYPE∞ score for NCSNv2, versus 19.8 for NCSN.The human-evaluation-based ranking differs from the FID ranking in the cited comparison.
  • Ablation results: FID results without denoising are qualitatively similar to results with denoising in the ablation comparison.The appendix reports this similarity directly for the additional ablation results.

C.4 Additional results on ablation studies

Ablation studies show that the proposed technique groups jointly improve samples, while training/test losses, nearest neighbors, and interpolations provide additional checks of model behavior.

  • Ablation studies: Removing any technique group from NCSNv2 leads to worse samples in the visual ablations.The groups are Technique 5, Techniques 1,2,4, and Technique 3.
  • Generalization checks: Training and test losses remain close throughout training, indicating that NCSNv2 does not simply memorize the training data.The paper uses these curves as evidence against overfitting.
  • Generalization checks: Nearest-neighbor comparisons use ℓ2 distance in pretrained InceptionV3 feature space and account for random horizontal flips.This procedure is applied across CIFAR-10, CelebA, LSUN, and FFHQ samples.
  • Interpolation: NCSNv2 interpolation results are reported for CelebA, LSUN church_outdoor, bedroom, tower, and FFHQ.The appendix includes interpolation figures at resolutions from 64 × 64 to 256 × 256.
  • Sample quality: Uncurated NCSNv2 samples are shown for CIFAR-10, CelebA, LSUN church_outdoor, bedroom, tower, and FFHQ.The reported sample resolutions range from 32 × 32 to 256 × 256.
Loading 2006.09011v2…