PaperlayerView exact revision on arXiv

Source-linked AI summary

PacGAN: The power of two samples in generative adversarial networks

Zinan Lin, Ashish Khetan, Giulia Fanti, Sewoong Oh

arXiv:1712.04086v3cs.LGcs.ITstat.ML

TL;DR

GANs often generate insufficiently diverse samples, and the reasons mode collapse occurs or existing remedies work remain poorly understood. The paper introduces packing, which lets discriminators classify multiple same-class samples using a binary-hypothesis-testing analysis. It reports that packing penalizes mode-collapsed generators and significantly improves mitigation on benchmark datasets.

  • Problem

    GANs often suffer mode collapse, while the reasons for its occurrence and the effectiveness of existing remedies remain poorly understood.

  • Method

    Packing modifies the discriminator to jointly classify m samples from the real or generated distribution, analyzed through binary hypothesis testing and product distributions.

  • Results

    Packing naturally penalizes generators with strong mode collapse, and benchmark experiments show significant improvement in mitigating mode collapse.

  • Takeaways & Limitations

    PacGAN provides a principled, low-overhead framework applicable to existing GANs for mitigating mode collapse.

  • Takeaways & Limitations

    The analysis does not sharply characterize the full effect of packing degree m on Jensen-Shannon divergence, leaving this as an open question.

Abstract

from arXiv · show

Generative adversarial networks (GANs) are innovative techniques for learning generative models of complex data distributions from samples. Despite remarkable recent improvements in generating realistic images, one of their major shortcomings is the fact that in practice, they tend to produce samples with little diversity, even when trained on diverse datasets. This phenomenon, known as mode collapse, has been the main focus of several recent advances in GANs. Yet there is little understanding of why mode collapse happens and why existing approaches are able to mitigate mode collapse. We propose a principled approach to handling mode collapse, which we call packing. The main idea is to modify the discriminator to make decisions based on multiple samples from the same class, either real or artificially generated. We borrow analysis tools from binary hypothesis testing---in particular the seminal result of Blackwell [Bla53]---to prove a fundamental connection between packing and mode collapse. We show that packing naturally penalizes generators with mode collapse, thereby favoring generator distributions with less mode collapse during the training process. Numerical experiments on benchmark datasets suggests that packing provides significant improvements in practice as well.

1 Introduction

GANs can generate realistic data but often suffer mode collapse, producing insufficiently diverse samples. PacGAN addresses the limited theoretical understanding of this problem by analyzing discriminators through binary hypothesis testing and packing multiple samples.

  • Mode collapse is the lack of diversity in generated samples, including missing modes or nearly identical outputs.Examples include omitting handwritten digits or generating series of nearly identical images.
  • Existing remedies modify architectures, loss functions, or optimization algorithms, but their relationship to mode collapse is not well understood.
  • The paper views the discriminator as performing binary hypothesis testing between real and generated distributions.
  • It formally defines mode collapse and represents the relationship between real and generated distributions using a two-dimensional mode collapse region.
  • Using m-th order product distributions, the analysis shows that packed discrimination penalizes strongly mode-collapsed generators and favors distributions with less mode collapse.
  • PacGAN applies packing to existing GANs by jointly classifying m concatenated samples as real or generated, and benchmark experiments show significant improvement in mitigating mode collapse.

2 PacGAN: A novel framework for mitigating mode collapse

PacGAN modifies only the discriminator of an existing GAN so it jointly classifies multiple same-class samples. Packing concatenates samples into a higher-dimensional discriminator input while retaining the generator, loss function, and training hyperparameters.

  • PacGAN starts from an existing GAN and retains its generator architecture and loss function while changing only the discriminator.
  • The packed discriminator maps m independently drawn samples from either the real or generated distribution to one joint soft label.
  • PacGAN(m) names the mother architecture and packing degree m; examples include PacGAN3 and PacDCGAN4.
  • The proposed architecture increases the discriminator input layer by a factor of m and adjusts first-layer connectivity to preserve the mother architecture.
  • For convolutional discriminators, packed images are stacked as a tensor with depth m, increasing the input depth and corresponding first convolutional-layer weights.
  • Training uses minibatches of packed samples, with jointly labeled real or generated groups treated as higher-dimensional features.
  • PacGAN keeps standard training hyperparameters unchanged and adds only a marginal computational overhead, while the exact complexity remains architecture-dependent.

3 Experiments

Experiments evaluate packing across synthetic mixtures, stacked MNIST, and CelebA-related settings using mode coverage, sample quality, and reverse KL metrics. PacGAN generally improves mode-collapse metrics with minimal architectural overhead, while packing's quality effects and parameter confounding receive explicit examination.

  • Experimental setup: PacGAN is evaluated against GAN, MD, DCGAN, VEEGAN, Unrolled GANs, and ALI on synthetic and real datasets.The experiments include 2D-ring, 2D-grid, stacked MNIST, and CelebA.
  • Metrics: The experiments measure recovered modes, high-quality samples, and reverse KL divergence to assess mode collapse.Mode counts are used on datasets with clear, known modes, while reverse KL evaluates generated distributions.
  • 3.1 Synthetic data experiments: PacGAN outperforms or matches baseline schemes on all three synthetic-data metrics, with increasing packing degree generally improving 2D-grid mode recovery.On 2D-ring, PacGAN2 already recovers almost all modes, leaving little extra benefit from larger packing factors.
  • 3.1.1 The effect of parameter size: 2D Grid: Parameter count appears to have little effect, while packing improves recovered modes and reverse KL from GAN through PacGAN2, PacGAN3, and PacGAN4.Packing does not appear to change the fraction of high-quality samples, and the experiments have large standard errors.
  • 3.2.2 Unrolled GAN [46] experiment: PacDCGAN finds more modes on average than Unrolled GANs with k = 10, with lower reverse KL divergences in both reported experiments.The authors interpret this as evidence that packing has a more pronounced effect than unrolling.
  • Additional comparisons: Compared with PacGAN4, MD uses substantially more discriminator parameters while achieving worse metrics in one setting and similar metrics in another.The reported parameter counts are 747 versus 1,226,317 and 2,213 versus 2,458,533 for PacGAN4 and MD, respectively.

4 Theoretical analyses of PacGAN

The paper formalizes mode collapse through a two-dimensional mode-collapse region and analyzes how packing changes discriminator loss under product distributions. The analysis shows that packing increasingly distinguishes and penalizes mode-collapsing generators, while its degree trades computational cost against distinguishability.

  • Mode-collapse region: The mode-collapse region represents a pair of target and generator distributions in two dimensions, abstracting away domain-specific geometry.It is defined using pairs (ε, δ) for which the generator assigns at most ε probability to a set receiving at least δ probability under the target.
  • Hypothesis-testing interpretation: The region is equivalent to a binary hypothesis-testing region, enabling data-processing-based proofs about how packing affects discriminator loss.This equivalence connects the region to ROC curves and type I and type II errors.
  • Product distributions: Packing m samples makes the discriminator distinguish product distributions P^m and Q^m, whose total variation loss naturally penalizes strong mode collapse.The packed discriminator therefore sees joint samples rather than individual samples, exposing differences that an unpacked discriminator can miss.
  • Evolution under packing: A mode-collapsing generator’s total variation distance increases more rapidly with packing degree m than a non-mode-collapsing generator’s distance.The analysis identifies mode-collapsing pairs as occupying the upper part of the total variation region under packing.
  • Packing degree: The packing degree m creates a trade-off between computational complexity and additional distinguishability.The paper presents m as a practitioner-controlled choice rather than a fixed universal value.
  • Separation and penalty: Packing separates generators with and without mode collapse, increasingly penalizing collapsed generators and rewarding generators with less mode collapse as m grows.For the illustrated families, packing with m ≥ 5 completely separates the two families.

5 Proofs of the main results

The proofs use mode collapse regions to convert distributional questions into geometric bounds, then propagate those bounds through product distributions. This yields bounds on packed total variation and identifies infeasible parameter regimes.

  • Geometric proof technique: The region perspective transforms measure-theoretic questions into geometric problems in a compact two-dimensional plane.The authors use this representation to enumerate distribution pairs with equal total variation and obtain tight proof bounds.
  • Total-variation bounds: For any pair with total variation τ, the mode collapse region can be sandwiched between canonical inner and outer regions parameterized by α.The corresponding canonical pairs use binary alphabets for inner regions and alphabet size three for outer regions.
  • Total-variation bounds: Region dominance is preserved under m-th order product distributions, producing lower and upper bounds on dTV(P^m, Q^m) from canonical pairs.The bounds minimize over α for the inner construction and use the outer construction directly.
  • Mode-collapse-constrained bounds: For (ε, δ)-mode-collapse pairs, the outer bound remains unchanged while the inner bound requires canonical constructions that depend on α and the parameter regime.When α exceeds 1 −(τδ/(δ −ε)), a second inner construction is used and the smaller resulting total variation is selected.
  • Feasibility regimes: When τ < δ −ε, no pair at total variation τ can satisfy (ε, δ)-mode collapse, so the constrained optimization reduces to the unconstrained one.For larger τ, the analysis gives additional inner and outer bounds using canonical regions and parameterized constructions.
  • Feasibility regimes: The feasible set is empty when τ exceeds max{(δ −ε)/(δ +ε), (δ −ε)/(2 −δ −ε)}, with the applicable threshold determined by whether δ +ε ≤1.The outer bound is represented by a hexagonal region whose edges encode the relevant mode-collapse constraints.

6 Related work

This section reviews GAN training, the challenges of mode collapse and instability, and prior architectural, objective, optimization, and regularization approaches. It positions packing as a simple discriminator modification with theoretical and empirical support, while identifying related methods outside the paper’s scope.

  • GAN training: GAN training learns generator weights so generated samples approximate an unknown data distribution using a discriminator or critic in an iterative procedure.The generator maps random noise to the target domain, while the discriminator measures how far generated samples are from target samples.
  • GAN training: The discriminator can provide local information for jointly moving generated samples closer to the target, which updates generator weights through back-propagation.The generator Jacobian with respect to its weights and a step size determine the gradient update.
  • Packing: Standard GAN discriminators decompose decisions across individual samples, motivating the question of whether jointly processing multiple samples improves mode-collapse mitigation.The paper answers affirmatively through numerical experiments and theoretical analyses connecting joint sample processing to mode collapse.
  • Related methods: MMD-based GAN discriminators also process image pairs, but whether they mitigate mode collapse theoretically or empirically is left as future work.Their two-sample tests use maximum mean discrepancy and kernel functions, differing from the proposed packing architecture.
  • Related challenges: Related mitigation strategies modify architectures, loss functions, optimization algorithms, or regularization, while instability remains an explicitly unaddressed challenge in this work.The theoretical results assume an optimal discriminator and infinitely many data samples.
  • Mode collapse: Mode collapse includes missing data modes, low probability on substantial support subsets, and mapping distant latent vectors to similar samples; its causes and interactions remain incompletely understood.Examples include omitted handwritten digits, missing lizards, restricted variation within a mode, and nearly identical translated images.
  • Packing: Packing uses a simple discriminator modification and is reported to reduce mode collapse theoretically and in practice, with comparisons to three main literature approaches.The paper presents packing as easier to implement, quantitatively strong in experiments, and supported by a principled analysis.

7 Discussion

The packing framework mitigates mode collapse with low overhead, but its broader theoretical scope and best architecture remain open questions.

  • Packing theoretically and empirically mitigates mode collapse with low overhead.
  • Extending the analysis to Jensen-Shannon divergence and Wasserstein distances remains an open question.
  • The appropriate packing architecture is unresolved, with permutation-invariant designs suggested as a possible direction.
Loading 1712.04086v3…