Bayesian GAN

Yunus Saatchi; Andrew Gordon Wilson

Source-linked AI summary

Bayesian GAN

Yunus Saatchi, Andrew Gordon Wilson

arXiv:1705.09558v3May 26, 2017stat.MLcs.AIcs.CVcs.LG

TL;DR

GANs can model rich data distributions but suffer from mode collapse and often require training interventions. Bayesian GAN marginalizes generator and discriminator weights with stochastic gradient Hamiltonian Monte Carlo, yielding interpretable, diverse representations and strong semi-supervised results. Its sampling-based approach improves robustness, though collapse is not fully eliminated.

Problem
GAN training can cause mode collapse and typically requires interventions such as feature matching, label smoothing, or mini-batch discrimination.
Method
Bayesian GAN places distributions over generator and discriminator weights and marginalizes them using stochastic gradient Hamiltonian Monte Carlo.
Results
Bayesian GAN improves accuracy over DCGAN, Wasserstein GAN, and an ensemble of 10 DCGANs, reaching 99.3% testing accuracy with 100 labelled MNIST examples.
Takeaways & Limitations
Exploring multimodal generator-weight posteriors produces diverse, interpretable representations and enables state-of-the-art semi-supervised performance with simple inference.
Takeaways & Limitations
Sampling does not fully eliminate collapse and can become trapped in sharply peaked modes, producing low-entropy samples.

Abstract

from arXiv · show

Generative adversarial networks (GANs) can implicitly learn rich distributions over images, audio, and data which are hard to model with an explicit likelihood. We present a practical Bayesian formulation for unsupervised and semi-supervised learning with GANs. Within this framework, we use stochastic gradient Hamiltonian Monte Carlo to marginalize the weights of the generator and discriminator networks. The resulting approach is straightforward and obtains good performance without any standard interventions such as feature matching, or mini-batch discrimination. By exploring an expressive posterior over the parameters of the generator, the Bayesian GAN avoids mode-collapse, produces interpretable and diverse candidate samples, and provides state-of-the-art quantitative results for semi-supervised learning on benchmarks including SVHN, CelebA, and CIFAR-10, outperforming DCGAN, Wasserstein GANs, and DCGAN ensembles.

1 Introduction

GANs offer powerful implicit generative modeling but can collapse onto a few examples and often require training interventions. Bayesian GAN addresses these issues by representing uncertainty over network weights and uses straightforward inference for unsupervised and semi-supervised learning.

GANs transform white noise through a generator while a discriminator learns to distinguish generated samples from true data.
Mode collapse can make GANs memorize a few training examples, while feature matching, label smoothing, and mini-batch discrimination are commonly used to stabilize training.
Representing broad, multimodal posteriors over network weights can capture multiple meaningful generators rather than a single compact solution.Different posterior modes may correspond to generators with distinct interpretations and can better model the data distribution.
Bayesian GAN proposes a simple probabilistic formulation that marginalizes generator and discriminator weights for unsupervised and semi-supervised learning.The approach uses stochastic gradient Hamiltonian Monte Carlo and avoids feature matching and other ad-hoc techniques.
The paper reports state-of-the-art semi-supervised performance across benchmarks including SVHN, MNIST, CIFAR-10, and CelebA.

2 Bayesian GANs

The Bayesian GAN places distributions over generator and discriminator weights, then uses posterior sampling and marginalization for unsupervised and semi-supervised learning. Its formulation connects generated samples, real data, and labeled examples through posterior inference and model averaging.

2 Bayesian GANs: The model places distributions over generator and discriminator weights, inducing distributions over possible generators and discriminators.Sampling generator weights and white noise produces candidate data samples from the induced generative distribution.
2 Bayesian GANs: Posterior inference iteratively samples conditional posteriors for the generator and discriminator weights.The generator posterior is influenced by discriminator-assigned data probabilities, while the discriminator posterior uses a classification likelihood and its parameter prior.
2 Bayesian GANs: Monte Carlo marginalization integrates over noise samples, although minibatch conditioning typically makes the discriminator approximation less accurate.The noise distribution permits efficient exact sampling, while the discriminator posterior is conditioned on minibatch data.
2 Bayesian GANs: With uniform priors and MAP optimization, the Bayesian formulation has the same local optima as the classical GAN, while posterior sampling retains broader multimodal uncertainty.The paper distinguishes marginalizing the posterior from approximating it with a point mass.
2 Bayesian GANs: Posterior samples can provide multiple generators and discriminator committees, which the paper uses to address GAN collapse and strengthen adversarial learning.Different generator samples can produce data with an appropriate level of entropy, while discriminator samples amplify the adversarial signal.
2 Bayesian GANs: The semi-supervised extension jointly uses unlabeled observations, labeled examples, generated samples, and posterior model averaging for test-label prediction.The discriminator assigns class probabilities to labeled data and reserves class 0 for generated samples; predictions average over discriminator posterior samples.

3 Posterior Sampling with Stochastic Gradient HMC

The Bayesian GAN uses SGHMC to sample posterior distributions over generator and discriminator weights. Its iteration alternates SGHMC updates using noise and data minibatches, with friction, learning-rate, and step-size assumptions.

Posterior sampling: SGHMC samples the posterior distributions over generator and discriminator weights using noisy gradient estimates that mix in the limit with many minibatches.The method applies HMC-style posterior sampling to both networks rather than optimizing each weight vector to a single point.
Algorithm assumptions: The algorithm uses α as SGHMC friction and η as the learning rate, while assuming discretization noise is dominated by friction, requiring small step sizes.
Sampling iteration: Each iteration samples generator noise, updates generator-weight samples for M SGHMC steps, and appends the resulting samples to the generator set.
Sampling iteration: The discriminator update samples noise and data minibatches, then runs M SGHMC steps to update its posterior sample set.

4 Experiments

Across synthetic and image benchmarks, BayesGAN improves distribution coverage and semi-supervised accuracy while producing diverse generator samples without the stabilization tricks required by DCGAN.

Synthetic dataset: BayesGAN better matches the synthetic data distribution than a regular GAN, while the regular GAN’s testing JSD eventually increases with training iterations.The comparison uses JSD estimated from kernel density estimates; good Bayesian performance persists when D = 500.
MNIST: 99.3% testing accuracy with 100 labeled MNIST examples matches a fully supervised method trained on all 50,000 examples.The 100 examples represent 0.2% of the training data.
MNIST: Posterior samples yield distinct, complementary generator styles, whereas six independently trained DCGANs produce homogeneous samples.The Bayesian samples vary in stroke thickness, crispness, and faintness, corresponding to different generator hypotheses.
Training and limitations: BayesGAN requires none of the minibatch discrimination, feature normalization, or discriminator-noise interventions used to make maximum-likelihood DCGAN converge.The authors attribute robustness to a Gaussian weight prior and MCMC exploration, while noting that some posterior samples still have low entropy.
CIFAR-10: On CIFAR-10, Bayesian GANs significantly reduce test error relative to alternatives, especially when the number of labeled examples is much smaller than the training set.The same experiments show meaningful qualitative variation across generators sampled from the posterior.

5 Discussion

The discussion argues that Bayesian GANs represent diverse, interpretable generator hypotheses and support semi-supervised learning where labels are costly. It positions sampling-based inference as an alternative to compact variational approximations and identifies marginal likelihood estimation as future work.

5 Discussion: Bayesian GANs capture diverse and complementary data representations that enable state-of-the-art semi-supervised performance with simple inference.The paper links rich multimodal generator-weight distributions to interpretable representations and benchmark performance.
5 Discussion: Semi-supervised learning matters because labels for high-dimensional natural data can be costly or unavailable, while providing a practical benchmark for unsupervised-learning advances.The paper cites human labeling and expensive instrumentation as sources of annotation cost.
5 Discussion: Sampling-based inference is suited to broad, multimodal neural-network posteriors, whereas variational approximations typically concentrate around a single mode.The discussion contrasts full-posterior exploration with overly compact single-mode approximations.
5 Discussion: Estimating a probabilistic GAN’s marginal likelihood remains future work for hyperparameter learning and principled comparison of GAN architectures.The paper also suggests combining Bayesian GANs with non-parametric Bayesian deep learning.

A.1 Rescaling conditional posteriors to accommodate mini-batches

The supplementary method rescales minibatch likelihoods so conditional-posterior updates remain consistent as dataset size changes. The likelihood should increasingly dominate the prior as more training data are available.

A.1 Rescaling conditional posteriors to accommodate mini-batches: Minibatch evaluations require likelihood rescaling in the conditional posteriors used by the Bayesian GAN updates.The same rescaling principle applies to the discriminator posterior and semi-supervised objectives.
A.1 Rescaling conditional posteriors to accommodate mini-batches: As the total number of training points N increases, the likelihood should dominate the prior.This scaling requirement motivates the conditional-posterior adjustment.

A.2 Additional Results

Additional results show how posterior samples yield distinct generator behaviors and how SG-HMC mixes and learns efficiently. Across CIFAR-10, SVHN, and CelebA, different posterior generator samples produce complementary qualitative outputs.

A.2 Additional Results: After about 1000 SG-HMC iterations, the sampler mixes reasonably well and learns the data distribution more efficiently per iteration than alternatives.Figure 4 plots test accuracy against iteration number.
A.2 Additional Results: Test accuracy is also plotted as a function of wall-clock time to assess runtime-dependent learning behavior.Figure 5 provides the time-based view without a stated outcome in the supplied passage.
A.2 Additional Results: Four posterior samples of generator weights produce qualitatively different CIFAR-10, SVHN, and CelebA generators with complementary properties.The figure compares samples from four generators across all three datasets.
A.2 Additional Results: Different posterior generator samples also yield qualitatively different CIFAR-10 outputs, indicating complementary aspects of the learned distribution.Figure 7 provides a larger CIFAR-10 sample set.
A.2 Additional Results: Additional samples from four posterior generators show qualitatively different SVHN outputs.The figure focuses on a larger set of SVHN samples.
A.2 Additional Results: Additional samples from four posterior generators show qualitatively different CelebA outputs.The figure focuses on a larger set of CelebA samples.