PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaGAN: Boosting Generative Models

Ilya Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, Bernhard Schölkopf

arXiv:1701.02386v2stat.MLcs.LG

TL;DR

GANs can produce realistic samples but may miss modes, leaving parts of the data distribution uncovered. AdaGAN iteratively adds generators trained on reweighted samples, creating a boosting-style mixture that focuses on hard examples. The paper shows convergence under stated conditions and reports that toy experiments address the missing-modes problem, while noting limitations of the resulting latent representation and empirical GAN objective.

  • Problem

    GANs are difficult to train and can miss modes, failing to produce examples from certain regions of the data distribution.

  • Method

    AdaGAN incrementally adds generators to an additive mixture by optimizing a generative-model criterion on reweighted data, with the framework extending to f-divergences and multiple generator types.

  • Results

    The mixture converges to the true distribution in a finite number of steps under optimal updates or at an exponential rate otherwise, while toy experiments show robust coverage of all data modes.

  • Takeaways & Limitations

    AdaGAN progressively covers missing modes by combining reweighted weak generators into a mixture, addressing a major practical issue in GAN training.

  • Takeaways & Limitations

    The resulting mixture lacks the smooth latent structure of a standard single-network GAN, limiting smooth interpolation in latent space.

Abstract

from arXiv · show

Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) are an effective method for training generative models of complex data such as natural images. However, they are notoriously hard to train and can suffer from the problem of missing modes where the model is not able to produce examples in certain regions of the space. We propose an iterative procedure, called AdaGAN, where at every step we add a new component into a mixture model by running a GAN algorithm on a reweighted sample. This is inspired by boosting algorithms, where many potentially weak individual predictors are greedily aggregated to form a strong composite predictor. We prove that such an incremental procedure leads to convergence to the true distribution in a finite number of steps if each step is optimal, and convergence at an exponential rate otherwise. We also illustrate experimentally that this procedure addresses the problem of missing modes.

1 Introduction

AdaGAN addresses GANs’ missing-modes problem by iteratively adding generators trained on reweighted data, forming an additive mixture that emphasizes examples the current mixture misses. The procedure is a boosting-style meta-algorithm applicable beyond GANs and supported by theoretical guarantees.

  • Method: AdaGAN automatically reweights training examples using a classifier that distinguishes true data from samples generated by the current mixture.Examples that the mixture fails to cover receive greater emphasis in the next training iteration.
  • Motivation: GANs can generate realistic data yet miss modes, producing insufficient variability in the learned distribution.Different random restarts may cover different modes, motivating procedures that combine multiple generators.
  • Scope: AdaGAN is a meta-algorithm rather than an improvement to a particular generator, and can be combined with GANs, VAEs, Gaussian mixture models, WGANs, and other models.The analysis is framed as minimizing an f-divergence for an additive mixture of distributions.
  • Method: Each iteration trains a new weak generator on reweighted data and adds it to the existing mixture with a chosen mixture weight.The overall model is updated as G_t = (1 − β_t)G_{t−1} + β_tG_c^t.
  • Relation to prior work: The resulting additive mixture differs from product-of-components boosting approaches because it remains directly sampleable while optimizing fixed-weight updates for general f-divergences.The paper contrasts its overall optimal update with prior infinitesimal-weight or log-domain procedures.

2 Minimizing f-divergence with Additive Mixtures

The paper formulates additive mixture construction as minimizing an f-divergence, derives reweighted targets for successive components, and analyzes when these updates improve and converge to the data distribution. Under suitable conditions, convergence can be exponential or finite, while practical progress depends on how well the new component is fitted.

  • 2.1 Preliminaries and notations: The framework minimizes an f-divergence between the real data distribution Pd and an additive mixture model Pmodel.Generative models are treated as distributions induced by transformations of a latent noise distribution, even when their densities are unavailable.
  • 2.2 Incremental Mixture Building: As the mixture improves, the next component is targeted toward a reweighted data distribution that emphasizes regions missed by the current mixture.The reweighting is introduced through an upper bound involving Df(Q ∥ Q0), where Q0 is computed from the original data distribution.
  • 2.3 Optimal Updates: The optimal next-component solution is independent of the chosen f-divergence and of the divergence order.The same solution applies across the considered f-divergences and when the arguments are exchanged.
  • 2.5 Convergence Analysis for Optimal Updates: With optimal components and a fixed mixture weight β, the objective decreases at least to (1−β)^T Df(Pg ∥ Pd) after T updates.This establishes an exponential convergence rate under the theorem’s conditions.
  • 2.5 Convergence Analysis for Optimal Updates: Finite-step convergence occurs if and only if the first mixture component satisfies a bounded density-ratio condition relative to Pd.When the condition holds, convergence takes at most −ln max(M, 1)/ ln(1 −β) steps.
  • 2.6 Weak to Strong Learnability: Strict improvement is guaranteed under a milder condition when β exceeds the mass of data missed by the current mixture, but stronger weak-learnability conditions may become harder as β decreases.The analysis also notes that sufficiently accurate approximation of the optimal component preserves improvement, while the required γ < β/4 condition can become increasingly demanding.

3 AdaGAN

AdaGAN incrementally builds a mixture by adding components trained on continuously reweighted data, emphasizing regions missed by the current mixture. The reweighting is estimated adversarially, and the procedure includes practical rules for mixture weights and an algorithmic implementation.

  • Incremental mixture building: Each iteration adds a component Q with weight β, forming (1 − β)Pg + βQ, where Q targets an optimal reweighted distribution.The target component is designed to correct deficiencies in the current mixture.
  • Adversarial reweighting: A discriminator between true data Pd and the current mixture Pg estimates the density ratio needed to construct the reweighted target.For Jensen-Shannon divergence, the optimal discriminator provides the relevant ratio transformation.
  • Computing training weights: On empirical training data, example weights are computed from discriminator values and normalized by selecting λ* through sorting and successive tests.The procedure determines the active set I(λ*) by sorting h(di) and testing candidate values.
  • Optimal updates: When β is sufficiently large, the optimal component can exactly match Pd, provided the GAN can approximate the target Q*.This exact matching condition is established by Lemma 4.
  • Mixture-weight choices: Because practical generators cannot match Q* perfectly, AdaGAN offers fixed, equal-weight, and adaptive heuristics for choosing β.Adaptive choices can use the fraction of examples whose density-ratio estimates exceed a threshold.
  • AdaGAN algorithm: Algorithm 3 sequentially chooses a mixture weight, updates example weights, trains a weak generator, and updates the overall mixture using a supplied GAN implementation.The meta-algorithm accepts a training sample and outputs a mixture generative model.

4 Experiments

The experiments evaluate whether AdaGAN's reweighting improves coverage of missing modes on toy Gaussian mixtures and MNIST. Boosting and ensembling outperform vanilla GAN, with AdaGAN showing faster convergence and lower variance, while MNIST results reveal underrepresented stroke patterns.

  • Experimental setup: AdaGAN is tested on toy datasets and MNIST to assess whether reweighting promotes diversity and covers missing modes.The experiments focus on coverage rather than the visual quality of individual samples.
  • Toy datasets: The toy targets are isolated mixtures of Gaussians whose number of modes varies to test performance under different mode counts.Component centers are sampled in a square, with variance adjusted so modes remain separated.
  • Compared methods: The comparison includes vanilla GAN, best-of-T restarts, independent ensembles, and sequential reweighting methods including AdaGAN.All approaches use the same underlying GAN implementation.
  • Evaluation: Coverage C measures the probability mass of the true distribution covered by the model's highest-density 95% region, with likelihood reported separately.Results use medians and 5%-95% intervals over 35 runs.
  • Toy results: Vanilla GAN fails to cover all modes, while ensemble and boosting methods significantly outperform vanilla GAN and best-of-T; AdaGAN converges faster with lower variance.Improvements appear after only one or two additional iterations, and vanilla GAN exhibits catastrophic failures in this setup.
  • MNIST results: On MNIST, reweighting repeatedly emphasizes digits with specific thick, thin, vertical, or diagonal strokes, suggesting underrepresented traits and more than ten dataset modes.The authors note that evaluating generative-model quality in this setting is unclear.

5 Conclusion

AdaGAN incrementally builds an additive mixture of generative models to address missing modes, with theoretical convergence guarantees and preliminary experimental support. The mixture improves coverage but sacrifices smooth latent structure.

  • AdaGAN incrementally adds generators trained on reweighted data to progressively cover modes of the true distribution.The procedure constructs an additive mixture by optimizing a GAN criterion on data reweighted toward undercovered regions.
  • Theoretical analysis gives conditions for convergence to the true distribution at an exponential rate or in finitely many steps.
  • Preliminary toy-data experiments indicate that AdaGAN addresses missing modes and robustly produces mixtures covering all data modes.
  • The resulting mixture lacks the smooth latent structure available in a single standard GAN.A partitioned latent representation may nevertheless represent discrete dataset aspects.

A Further details on toy experiments

Toy experiments compare AdaGAN using usual and unrolled GANs, showing that the meta-algorithm can use different underlying GAN procedures. Unrolled-GAN steps require more computation.

  • AdaGAN was evaluated with both a usual GAN and an unrolled GAN using multiple training objectives.Figure 4 compares the architectures and objectives across the toy experiments.
  • AdaGAN works with all underlying GAN algorithms tested in the toy experiments.
  • Figure 4 compares coverage C of the true data by the model distribution across iterations T.The top row uses the usual GAN and the bottom row uses the unrolled GAN.
  • One AdaGAN step with an unrolled GAN takes roughly the computation time of three steps with a usual GAN.The unrolled GAN uses five unrolling steps in these experiments.

B Further details on MNIST/MNIST3 experiments

MNIST experiments use convolutional and deconvolutional GAN architectures with a 100-dimensional latent space and a classifier-based reweighting procedure. Reweighting emphasizes digits with specific strokes that appear underrepresented.

  • AdaGAN was run on 28x28-pixel MNIST images using convolutional and deconvolutional networks with batch normalization and leaky ReLU.
  • The generator maps a 100-dimensional latent input through fully connected and deconvolutional layers to image features.
  • The discriminator processes 28x28x1 inputs through convolutional layers producing progressively smaller feature maps.
  • Training used Adam with β1 = 0.5, two generator steps per discriminator step, and separate learning rates for G, D, and the reweighting classifier C.
  • Reweighting focuses on MNIST digits with specific strokes, including thick, thin, vertical, and diagonal traits.The observations suggest more than 10 modes in MNIST, though the quality of generative models in this setting is difficult to evaluate.

C.1 Proof of Theorem 1

The proof of Theorem 1 relies on preliminary lemmas concerning a scalar function and comparisons between divergences of mixtures.

  • The proof introduces one lemma about determining the constant λ and another about comparing divergences of mixtures.
  • Lemma 5 applies to distributions P and Q with γ ∈ [0, 1] and λ ∈ R.
  • The function in Lemma 5 is nonnegative, convex, nondecreasing, and satisfies g(λ) ≤ λ.The lemma also characterizes its right derivative.

The equation

The section establishes properties of the reweighting equation and uses convexity to prove the optimality of the next mixture component. It also identifies conditions ensuring existence and uniqueness of the relevant parameter.

  • Existence and uniqueness: The reweighting equation has a solution λ∗ in [1 −γ, 1], and λ∗ is unique when γ < 1.The proof establishes existence through continuity and convexity, then rules out constant intervals away from zero.
  • Optimal component: The construction defines the next component through a clipped density, with λ∗ selected so the resulting expression is a valid distribution.The proof first establishes that λ∗ lies in the required interval and then invokes the convex comparison inequality.
  • Convexity bound: Convexity yields E[f(max(c, Y))] ≤ E[f(X + Y)] − E[X(f′(Y) − f′(c))+] ≤ E[f(X + Y)].The bound follows by decomposing the expectation according to whether Y is below or above c.
  • Optimal component: Substituting X = βdQ/dPd and Y = (1 −β)dPg/dPd into the bound proves that Q∗β is optimal among candidate distributions.Both transformed quantities have expectation 1 with respect to Pd, allowing the lemma to be applied.

C.2 Proof of Theorem 2

The proof analyzes a constrained minimization through auxiliary convex functions and characterizes when the parameter λ† exists or is unique. It then transfers this solution to the mixture-component optimization.

  • Auxiliary function: The auxiliary function h is convex and non-increasing, which controls the number of solutions to the defining equation for λ†.These shape properties follow from convexity and monotonicity of the positive-part function under integration.
  • Existence of λ†: The defining equation has no solutions if Δ > 1 −γ, one solution λ† ≥ 1 if Δ < 1 −γ, and possibly infinitely many or no solutions when Δ = 1 −γ.The boundary case depends on whether h reaches its limiting value at a finite point.
  • Convex comparison: Lemma 8 supplies a lower bound for convex f when X ≤ Y almost surely and E[min(c, Y)] = E[X].The proof decomposes the expectation according to whether Y is below or above c.
  • Mixture optimization: Applying Lemma 9 and changing variables converts the constrained minimization into the desired optimization over mixture components.The resulting solution has a form independent of f, so optimizing the auxiliary divergence gives the same solution for Df.
  • Assumption: The proof assumes that a constant c satisfying the lemma’s expectation condition exists; this is not guaranteed in general.Under this assumption, the argument identifies a unique c in [1, ∞) satisfying the relevant equation.

D f-Divergences

This section writes the Jensen–Shannon divergence in terms of the two distributions and their mixture, using the standard symmetric logarithmic form.

  • Definition: The Jensen–Shannon divergence combines logarithmic terms weighted by p(x) and q(x), with denominators involving p(x) + q(x).The displayed expression includes contributions from both distributions relative to their mixture.
  • Definition: The divergence expression uses the binary-mixture normalization through terms such as 2q(x)/(p(x) + q(x)) and 2p(x)/(p(x) + q(x)).These normalized ratios appear inside the logarithms for the q- and p-weighted terms.

E Additional experimental results

The additional experiments compare several reweighting heuristics across mixture sizes and show that tuning matters. The simple β = 1/t rule performs better than the tuned alternatives after few iterations.

  • Compared heuristics: The experiments compare constant β, previous-generator top-fraction reweighting, and mixture-discriminator top-fraction reweighting.The latter two retain the highest-scoring examples according to a discriminator, while constant β explores a range of values.
  • Results: When properly tuned, each reweighting scheme outperforms the baselines and has similar performance over few iterations.These alternatives nevertheless introduce an additional parameter that must be tuned.
  • Results: The simple β = 1/t heuristic is better than the alternative reweighting schemes despite requiring no additional parameter.The comparison is explicitly made after noting the tuning burden of the alternatives.
  • Evaluation: The reported scores are medians with 5%–95% intervals over 35 runs, covering both coverage C and log likelihood.The tables use the interval to describe algorithmic stability rather than a conventional confidence interval.
  • Gaussian-mixture experiment: For Gaussian mixtures, ensemble and boosting approaches significantly outperform vanilla GAN with three iterations, while boosting reaches optimal coverage faster.Three iterations correspond to two additional mixture components.
Loading 1701.02386v2…