PaperlayerView exact revision on arXiv

Source-linked AI summary

Triple Generative Adversarial Nets

Chongxuan Li, Kun Xu, Jun Zhu, Bo Zhang

arXiv:1703.02291v4cs.LGcs.CV

TL;DR

Existing SSL GANs can face incompatible classification and generation objectives and lack semantic control over generated samples. Triple-GAN separates generation, discrimination, and classification with compatible utilities and conditional image-label modeling, achieving state-of-the-art classification among deep generative models while disentangling classes and styles.

  • Problem

    Existing SSL GANs may not optimize generators and classifiers simultaneously and cannot control generated-sample semantics because two-player discriminators combine incompatible roles.

  • Method

    Triple-GAN uses a generator, discriminator, and classifier, with conditional networks modeling p(y|x) and p(x|y) and a discriminator judging image-label pairs.

  • Results

    Triple-GAN achieves state-of-the-art classification among deep generative models and simultaneously disentangles classes and styles for class-conditional interpolation.

  • Takeaways & Limitations

    Compatible utilities enable the classifier and generator to reach their respective optima while supporting class-specific sampling and smooth latent-space transfer.

  • Takeaways & Limitations

    The method assumes samples from both p(x) and p(y) can be easily obtained, with p(y) treated as the labeled-data label distribution in the experiments.

Abstract

from arXiv · show

Generative Adversarial Nets (GANs) have shown promise in image generation and semi-supervised learning (SSL). However, existing GANs in SSL have two problems: (1) the generator and the discriminator (i.e. the classifier) may not be optimal at the same time; and (2) the generator cannot control the semantics of the generated samples. The problems essentially arise from the two-player formulation, where a single discriminator shares incompatible roles of identifying fake samples and predicting labels and it only estimates the data without considering the labels. To address the problems, we present triple generative adversarial net (Triple-GAN), which consists of three players---a generator, a discriminator and a classifier. The generator and the classifier characterize the conditional distributions between images and labels, and the discriminator solely focuses on identifying fake image-label pairs. We design compatible utilities to ensure that the distributions characterized by the classifier and the generator both converge to the data distribution. Our results on various datasets demonstrate that Triple-GAN as a unified model can simultaneously (1) achieve the state-of-the-art classification results among deep generative models, and (2) disentangle the classes and styles of the input and transfer smoothly in the data space via interpolation in the latent space class-conditionally.

1 Introduction

Existing SSL GANs struggle to optimize classification and generation simultaneously and cannot control generated semantics because a single discriminator combines incompatible roles. Triple-GAN addresses these issues with three players, compatible objectives, and joint image-label modeling.

  • Existing SSL GANs may not optimize the generator and classifier simultaneously, while their generators cannot control generated-sample semantics.
  • Feature matching favors classification but produces distinguishable samples, whereas minibatch discrimination favors realistic generation but predicts labels inaccurately.
  • Two-player formulations create incompatible objectives because one discriminator must identify fake samples and predict labels simultaneously.
  • Triple-GAN uses a classifier and conditional generator to model p(y|x) and p(x|y), while one discriminator distinguishes real from generated image-label pairs.
  • Compatible utilities make a good classifier support a good generator and vice versa, while label information helps generate correct image-label pairs.
  • Triple-GAN achieves state-of-the-art classification among deep generative models and disentangles classes and styles for class-conditional interpolation.

2 Related Work

Related work spans directed deep generative models, GANs for realistic image synthesis, inference networks, and semi-supervised methods for partially labeled data. Triple-GAN is positioned alongside these approaches as a model combining classification and conditional generation.

  • Directed deep generative models include VAEs, GMMNs, and GANs, whose criteria are systematically compared in prior work.
  • GAN-based image-generation methods include LAP-GAN for progressive upscaling and DCGAN for realistic natural images.
  • InfoGAN learns explainable latent codes through variational mutual-information maximization, while ALI uses an inference network to approximate latent posteriors.
  • Semi-supervised approaches handle missing labels using latent variables, auxiliary variables, denoising autoencoders, or categorical discriminators.

3 Method

Triple-GAN models semi-supervised classification and class-conditional generation with three players that jointly characterize image-label distributions. Its compatible utilities make the classifier and generator converge to the true joint distribution at a unique global optimum.

  • Problem setting: Triple-GAN targets semi-supervised label prediction and class-conditional sample generation from partially labeled data.The setting requires modeling uncertainty over both inputs and labels through their joint distribution.
  • Three-player model: The game uses p(x,y) = p(x)p(y|x) = p(y)p(x|y) to estimate the conditional distributions needed for classification and generation.The mixture distribution combines the classifier- and generator-defined joint distributions with weight α.
  • Three-player model: The classifier models p(y|x), the generator models p(x|y), and the discriminator distinguishes real from generated image-label pairs.The discriminator receives pairs from the true distribution and from both conditional networks.
  • Theoretical analysis: Adding supervised cross-entropy resolves non-uniqueness and yields a unique global optimum for the classifier and generator.The supervised loss is equivalent to the KL-divergence between pc(x,y) and p(x,y).
  • Theoretical analysis: At equilibrium, p(x,y) = pg(x,y) = pc(x,y), so both networks converge to the true data distribution rather than competing.The same equilibrium also implies matching input and label marginals for p, pc, and pg.
  • Pseudo discriminative loss: The pseudo discriminative loss trains the classifier on generator-produced labeled samples while preserving the global equilibrium.It is equivalent to minimizing D_KL(pg(x,y)||pc(x,y)).

4 Practical Techniques

Triple-GAN adds practical procedures for partially labeled data, addressing discriminator overfitting, classifier regularization, and nondifferentiable label sampling. These techniques may bias the theoretical solution but perform well empirically on difficult SSL tasks.

  • Training considerations: Practical techniques are introduced for challenging semi-supervised learning tasks, despite possible theoretical bias.The implementation methods are intended to improve empirical training behavior.
  • Discriminator training: Using classifier-generated pseudo-labeled unlabeled data as discriminator positives reduces memorization of the small labeled set and generator mode collapse.This changes the discriminator target from pure p to a mixture of pc and p, which is acceptable when pc is close to p.
  • Classifier regularization: The confidence loss encourages confident, label-balanced predictions on unlabeled data, while consistency loss penalizes predictions that vary across noise realizations.Confidence loss is the default except on CIFAR10.
  • Gradient estimation: REINFORCE is used to address nondifferentiable feedback when optimizing the classifier through expectations over discrete class labels.Direct label sampling prevents discriminator feedback from being differentiable with respect to classifier parameters.

5 Experiments

Experiments on MNIST, SVHN, and CIFAR10 evaluate Triple-GAN for semi-supervised classification, generation, disentanglement, and class-conditional interpolation. Triple-GAN consistently improves classification and produces semantically controlled, smoothly varying samples.

  • Classification: Triple-GAN achieves state-of-the-art classification results across MNIST, SVHN, and CIFAR10, substantially outperforming strong competitors on SVHN and CIFAR10.The evaluation uses partially labeled datasets with 100, 1,000, and 4,000 labels, respectively.
  • Classification: Triple-GAN consistently outperforms Improved-GAN on MNIST with 20, 50, and 200 labeled samples, with larger gains as labels become scarcer.This pattern suggests the pseudo discriminative loss is particularly useful in low-label settings.
  • Classification: On SVHN, removing the classifier-free generator–classifier interaction raises error above 10%, while removing the pseudo discriminative loss yields about 7.8% error versus 8.11% for Improved-GAN.The complete model is about 2 percentage points better than the version without the pseudo discriminative loss.
  • Generation: Triple-GAN generates clearer SVHN digits than Improved-GAN with feature matching and produces fewer meaningless or repeated samples.On CIFAR10, its inception score is 5.08 ± 0.09 versus 3.87 ± 0.03 for Improved-GAN without minibatch discrimination.
  • Generation: Triple-GAN disentangles class identity from latent style factors such as scale, intensity, orientation, and color using much less label information than fully supervised methods.Samples remain realistic within a specified class while latent variables encode meaningful physical factors.
  • Generation: Class-conditional latent interpolation transitions smoothly between samples with different visual factors without losing label semantics.The experiment linearly interpolates between two latent vectors and maps them to data while holding each class label fixed.

6 Conclusions

Triple-GAN is a unified three-player framework for semi-supervised learning that uses compatible utilities to address limitations of existing GAN-based methods.

  • Triple-GAN comprises a generator, discriminator, and classifier for semi-supervised learning with compatible utilities.
  • The framework enables both the classifier and generator to achieve their respective optima from a game-theoretic perspective.
  • Triple-GAN enables the generator to sample data from a specified class.
  • On MNIST, SVHN, and CIFAR10, Triple-GAN achieves state-of-the-art classification results among deep generative models while disentangling styles and classes.
  • The model transfers smoothly at the data level through interpolation in the latent space.

A Detailed Theoretical Analysis

The theoretical analysis characterizes the game’s equilibrium through joint-distribution matching and shows that the classifier and generator converge to the data distribution under the designed utilities.

  • The discriminator’s optimality condition yields a mixture distribution pα(x, y) := (1 −α)pg(x, y) + αpc(x, y).
  • The global minimum of V(C, G) occurs if and only if the data joint distribution equals the mixture distribution, p(x, y) = pα(x, y).
  • At this equality, the marginal distributions of the data, classifier, and generator match for both images and labels.
  • The equilibrium of the modified utility is achieved if and only if p(x, y) = pg(x, y) = pc(x, y).
  • Minimizing the added regularization term is equivalent to minimizing DKL(p(x, y)||pc(x, y)), which reaches zero exactly when the joint distributions match.
  • Adding divergences between joint, conditional, or marginal distributions as regularization does not change the global equilibrium.

B Unconditional Generation

Triple-GAN improves unconditional image generation over feature-matching Improved-GAN baselines and produces meaningful samples conditioned on specified CIFAR10 classes.

  • Triple-GAN outperforms feature-matching Improved-GAN on generating indistinguishable samples for MNIST and CIFAR10.
  • Triple-GAN generates meaningful images in specific CIFAR10 classes.

D Disentanglement and Interpolation on the MNIST dataset

On MNIST, Triple-GAN disentangles class and style and supports class-conditional interpolation with smooth data-level transfer and clear semantics.

  • Triple-GAN disentangles class and style on MNIST.
  • Triple-GAN performs class-conditional interpolation that transfers smoothly at the data level with clear semantics.

E Convergence Speed

Triple-GAN converges at a speed comparable to Improved-GAN on SVHN despite using an additional network, achieving good results within tens of epochs. Its learning curve can oscillate, while applying pseudo discriminative loss later substantially reduces test error.

  • Triple-GAN’s convergence speed is at least comparable to Improved-GAN on SVHN despite using one additional network.Both models use default settings.
  • Triple-GAN can obtain good results within tens of epochs, although its learning curve is oscillatory.The oscillation may result from larger gradient variance caused by discrete variables.
  • Applying pseudo discriminative loss at epoch 200 significantly reduces test error within the following 100 epochs.

F Detailed Architectures

The paper provides dataset-specific Triple-GAN architectures for MNIST, SVHN, and CIFAR10. The listed components cover the classifier, discriminator, and generator, including convolutional, normalization, pooling, deconvolutional, MLP, sigmoid, tanh, and Softmax operations.

  • Detailed Triple-GAN architectures are provided separately for MNIST, SVHN, and CIFAR10 in Tables 3, 4, and 5.
  • The architecture descriptions organize components under three players: classifier C, discriminator D, and generator G.
  • The listed networks use convolutional layers, leaky ReLU activations, normalization, and global pooling in the architecture specifications.
  • The generator specifications include a stride-2 deconvolutional layer, while classifier outputs include a 10-class Softmax.
  • The specifications also include MLP outputs with sigmoid or tanh activations, alongside weight normalization in several components.
Loading 1703.02291v4…