Adversarial score matching and improved sampling for image generation

1 INTRODUCTION

• Song and Ermon's improved DSM-ALS produced more visually appealing images, but its reported FID did not reflect that improvement.This mismatch motivates the paper's use of final-sample denoising.Source: §1 · pp. 1–2
• DSM-ALS has high sample diversity through ALS, but its higher-resolution image perceptual quality has been inferior to GAN-generated images.GANs, in turn, often have trouble producing data from all possible modes.Source: §1 · pp. 1–2
• The paper proposes CAS as a more stable alternative to ALS, final-sample denoising to improve FID, and a hybrid objective combining score matching with adversarial training.The hybrid objective encourages the score network's expected denoised samples to appear realistic to a discriminator.Source: §1 · p. 2
• The proposed contributions bring DSM-ALS into range of state-of-the-art image generation on CIFAR-10 and LSUN-church by FID.The paper also reports that Ho et al.'s architecture significantly improves sample quality over RefineNet.Source: §1 · p. 2

2.1 DENOISING SCORE MATCHING

• DSM trains a noise-conditioned score network to approximate the log-density gradient using noisy surrogates of the data distribution.Training samples noise levels uniformly from a geometric progression, with Gaussian corruption at each level.Source: §2.1 · p. 2

2.2 ANNEALED LANGEVIN SAMPLING

• ALS addresses Langevin sampling's difficulty exploring distributions with widely separated modes by beginning at high noise and progressively reducing it.The annealing is intended to provide mode coverage and convergence to the data distribution.Source: §2.2 · p. 3

2.3 EXPECTED DENOISED SAMPLE (EDS)

• The expected denoised sample is the score-based estimate of the real image given its corrupted version, and can serve as the final Langevin sample.The paper reports that final-sample denoising consistently decreases FID and brings samples closer to the assumed data manifold.Source: §2.3 · p. 3

3.1 INCONSISTENCIES IN ALS

• ALS does not maintain its assumed geometric noise schedule, even when annealing converges.With an optimal score, the noise variance remains greater than σ_t² at every step; finite-step residual noise carries into later iterations, and non-optimal scores are expected to leave still more noise.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 ALGORITHM

• CAS ensures that noise follows the prescribed schedule for any sampling step size and number of steps.With an optimal score, its noise variance equals σ_t² at every step, for any number of steps in the schedule.Source: §3.2 · p. 4
• The CAS proposition assumes initialization with a data image corrupted by noise; the authors say pure-noise initialization is practically indistinguishable when σ₀ is set sufficiently high.They set σ₀ to the maximum Euclidean distance between training-data pairs.Source: §3.2 · p. 5

4 BENEFITS OF THE EDS ON SYNTHETIC DATA AND IMAGE GENERATION

• On both synthetic 2D datasets, taking the EDS brings generated samples much closer to the real data manifold.The figure distinguishes real data points as circles and generated points as crosses.Source: Figure 2 · p. 5, §4 · p. 5
• Denoising is critical to ALS's CIFAR-10 FID, while denoised CAS maintains a low score across a wider range of sampling step sizes.With CAS, denoised and final-sample FIDs are approximately equal at small step sizes, and their gap is smaller than ALS's at larger step sizes.Source: §4 · pp. 5–6, Figure 3 · p. 6
• Denoising resolves the mismatch between improved visual quality and unimproved FID reported for Song and Ermon (2020).The paper notes that the improvement is not perceptible to the human eye.Source: §4 · pp. 5–6

5 ADVERSARIAL FORMULATION

• The score network minimizes score-matching loss while encouraging its denoised samples to be judged real by a discriminator.The authors motivate this with the poor correlation between L2 loss and human perception of image quality.Source: §5 · p. 6
• Alternating gradient descent trains the discriminator for a set number of steps at each score-function update.The experiments use the Least Squares GAN formulation, which performed best among the tested choices.Source: §5 · p. 6
• The adversarial objective acts on the expected denoised sample Hθ, derived from the score network, rather than a standard autoencoder output.The authors note its similarity to adversarial autoencoders, with a denoising autoencoder in place of a standard autoencoder.Source: §5 · p. 6
• Because GANs favor quality over diversity, the authors test whether the hybrid objective reduces the diversity of samples produced by ALS.They address this concern with experiments on 3-StackedMNIST.Source: §5 · p. 7

6.1 ABLATION STUDY

• The ablation varied adversarial training, Consistent Annealed Sampling, and denoising across CIFAR-10 and LSUN-Churches, using Song and Ermon (2020) and Ho et al. (2020) score-network architectures.The hybrid approach used an unconditional BigGAN discriminator.Source: §6.1 · p. 7
• Denoising and increasing nσ from 1 to 5 always improved FID across the reported ablations.The experiments covered CIFAR-10 and LSUN-Churches with the Song and Ermon (2019) architecture, and CIFAR-10 with the Ho et al. (2020) architecture.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 7
• 3.65 (non-adversarial) and 4.02 (adversarial) FID on CIFAR-10 with the Ho et al. (2020) architecture were close to its reported 3.17.These scores used 50k samples; the score network was trained for 300k iterations, versus more than 1 million for Ho et al. (2020).Source: Table 3 · p. 7, §6.1 · p. 8
• Adversarial training improved results with the Song and Ermon (2019) architecture, but not on denoised samples with the Ho et al. (2020) architecture.The authors hypothesize that the discriminator architecture limits the latter result; they still report advantages for simpler architectures.Source: §6.1 · p. 8
• Consistent sampling outperformed non-consistent sampling on CIFAR-10 at nσ = 1, the quickest sampling setting.Source: §6.1 · p. 8

6.2 NON-ADVERSARIAL AND ADVERSARIAL SCORE NETWORKS HAVE EQUALLY HIGH DIVERSITY

• On 3-StackedMNIST, all 1,000 modes were covered and the hybrid objective did not hurt diversity, although KL divergence remained inferior to PACGAN.Mode coverage and KL divergence were evaluated from predicted modes of a pretrained MNIST classifier.Source: §6.2 · p. 8, Table 4 · p. 8

7 CONCLUSION

• Consistent Annealed Sampling ensures the expected geometric progression of noise and brings final samples closer to the data manifold.The paper also describes extracting the expected denoised sample to further improve final Langevin samples, and proposes a hybrid GAN and score-matching approach.Source: §7 · p. 8
• Score matching performs better than most GANs and on par with state-of-the-art GANs.The authors also report equal coverage of all 1000 modes for adversarial and non-adversarial variants on StackedMNIST, alongside lower FIDs for their other adversarial image models.Source: §7 · pp. 8–9
• The authors identify scaling to larger batch sizes on high-resolution images as future work.They cite GAN results at batch sizes of 2048 or more as motivation for this direction.Source: §7 · p. 9

A BROADER IMPACT

• High-resolution generation is constrained by the need for TPUs or clusters of 8+ GPUs, and adding a discriminator further increases training load.The authors attribute the main cost to the architectures used in the paper.Source: A BROADER IMPACT · p. 13

B EXPERIMENTS DETAILS

• Sampling learning rates are reported for non-adversarial and adversarial score matching; the experiments use batch size 128, with 300k CIFAR-10 and 200k LSUN-Churches training iterations.EMA settings were .9999 for CIFAR-10 and .999 for LSUN-Churches; sampling step sizes were selected by denoised FID.Source: Table 5 · p. 13, B EXPERIMENTS DETAILS · p. 13
• The adversarial experiments swept loss functions, discriminator steps, optimizer parameters, and λ; LSGAN yielded the best FID among the tested losses.The reported settings use λ = 1, two discriminator steps on CIFAR-10, and one on LSUN-Churches.Source: B EXPERIMENTS DETAILS · pp. 13–14

C SUPPLEMENTARY EXPERIMENTS

• With four V100 GPUs, training the 256×256 FFHQ models under the prior setting was impossible; the reduced model performed poorly.The authors suspect the adversarial model's poor performance was related to the batch size of 32, below BigGAN's usual 2048-image batches at this resolution.Source: C SUPPLEMENTARY EXPERIMENTS · p. 14

D OPTIMAL CONDITIONAL SCORE FUNCTION

• For the conditional case, the loss minimizer equals the gradient of the log corrupted-data distribution at each noise level.The passage identifies this as the optimal conditional score function.Source: D OPTIMAL CONDITIONAL SCORE FUNCTION · p. 14

E OPTIMAL UNCONDITIONAL SCORE FUNCTION

• The appendix derives the unconditional optimum using an expectation over noise levels, with p(σ) chosen as a discrete uniform distribution.The derivation uses calculus of variations; the choice of p(σ) is stated to be unimportant for this derivation.Source: E OPTIMAL UNCONDITIONAL SCORE FUNCTION · pp. 14–15
• For a Dirac data distribution, the expected denoised sample is x0; the unconditional score replaces a noise level with the harmonic mean of noise levels.The approximation becomes inaccurate when σ_i is far from its mean: it overestimates the score at large noise and underestimates it at small noise.Source: E OPTIMAL UNCONDITIONAL SCORE FUNCTION · p. 15

• At every Langevin step, the noise-component variance remains above σ_t², even with the optimal score.Source: F ALS NON-GEOMETRIC PROOF · pp. 15–16
• Under conditions generally respected in practice, the noise variance decreases monotonically but converges to a value greater than σ_t².Source: F ALS NON-GEOMETRIC PROOF · p. 16
• With a geometrically decreasing noise schedule, the sample-noise standard deviation cannot follow the prescribed schedule, even given an optimal score.Source: F ALS NON-GEOMETRIC PROOF · p. 16

G CAS GEOMETRIC PROOF

• CAS keeps the sample’s noise variance equal to σ_i² at each step; with a geometric schedule, the noise standard deviation follows that schedule.Source: G CAS GEOMETRIC PROOF · pp. 16–17

H UPDATE RULE

• For a noise-conditional score function, the sampling updates in Algorithms 1 and 2 are equivalent to the stated update rules.Source: H UPDATE RULE · p. 17

I INCEPTION SCORE (IS)

• Denoising improves FID but not Inception Score; the authors advise against using IS to compare score-matching models within or across models.They attribute this to denoising confounding the textures IS uses to distinguish classes.Source: I INCEPTION SCORE (IS) · p. 17

J UNCURATED SAMPLES

• The cited CIFAR-10 figures depict adversarial and non-adversarial samples under both consistent and non-consistent sampling conditions, with nσ = 1 in each of the four figure captions.Source: Figure 4 · p. 18, Figure 5 · p. 18, Figure 8 · p. 20
• Uncurated LSUN-Churches figures show adversarial and non-adversarial samples, with consistent and non-consistent sampling at nσ=1.Source: Figure 12 · p. 22, Figure 13 · p. 22, Figure 16 · p. 24
• Uncurated LSUN-Churches figures also show adversarial and non-adversarial samples, with consistent and non-consistent sampling at nσ=5.Source: Figure 14 · p. 23, Figure 15 · p. 23, Figure 18 · p. 25
• Figures using the Ho et al. (2020) network architecture show CIFAR-10 adversarial and non-adversarial samples at nσ=1 in both consistent and non-consistent settings, and at nσ=5 only in the non-consistent setting.Source: Figure 20 · p. 26, Figure 21 · p. 26, Figure 22 · p. 27