PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks

Jost Tobias Springenberg

arXiv:1511.06390v2stat.MLcs.LG

TL;DR

CatGAN addresses learning discriminative classifiers from unlabeled or partially labeled data. It combines mutual-information-based categorical prediction with adversarial robustness, achieving competitive semi-supervised image-classification performance while producing visually faithful generated images.

  • Problem

    Learning discriminative classifiers from unlabeled or partially labeled data requires exploiting data structure without known true class assignments.

  • Method

    CatGAN combines a neural-network classifier maximizing mutual information with an adversarial generator whose examples enforce classifier robustness.

  • Results

    CatGAN achieves competitive state-of-the-art semi-supervised image-classification performance, while its jointly learned generator produces images with high visual fidelity.

  • Takeaways & Limitations

    The framework supports robust unsupervised and semi-supervised learning and connects adversarial classification with discriminative clustering.

  • Takeaways & Limitations

    The standard GAN objective can be unstable and can cause the generator to collapse onto one data mode or switch between modes.

Abstract

from arXiv · show

In this paper we present a method for learning a discriminative classifier from unlabeled or partially labeled data. Our approach is based on an objective function that trades-off mutual information between observed examples and their predicted categorical class distribution, against robustness of the classifier to an adversarial generative model. The resulting algorithm can either be interpreted as a natural generalization of the generative adversarial networks (GAN) framework or as an extension of the regularized information maximization (RIM) framework to robust classification against an optimal adversary. We empirically evaluate our method - which we dub categorical generative adversarial networks (or CatGAN) - on synthetic data as well as on challenging image classification tasks, demonstrating the robustness of the learned classifiers. We further qualitatively assess the fidelity of samples generated by the adversarial generator that is learned alongside the discriminative classifier, and identify links between the CatGAN objective and discriminative clustering algorithms (such as RIM).

1 INTRODUCTION

CatGAN addresses learning nonlinear classifiers and discovering categories from unlabeled or partially labeled data by combining discriminative classification with adversarial generation. It targets class-relevant structure rather than reconstruction of every input detail.

  • Learning nonlinear classifiers from unlabeled or partially labeled data remains a longstanding machine-learning problem.
  • Unsupervised learning assumes that structure in p(x) contains information about the unknown conditional label distribution p(y|x).
  • Existing approaches include generative clustering methods that model p(x) and discriminative methods that directly group unlabeled data.
  • Generative models and autoencoders often support classification pretraining, but reconstruction preserves all input information rather than only class-predictive information.
  • CatGAN classifiers maximize mutual information between inputs and predicted categorical labels while becoming robust to adversarially generated bogus examples.

2 GENERATIVE ADVERSARIAL NETWORKS

GANs formulate learning as a two-player game between a discriminator distinguishing real from generated inputs and a generator producing inputs intended to fool it. Differentiable models can optimize this game through alternating stochastic-gradient updates.

  • GAN training uses a discriminator D to distinguish real dataset inputs from samples generated by G.
  • The generator G maps random noise vectors z to generated inputs.
  • The discriminator predicts the probability that an input belongs to the real dataset using a sigmoid function of D(x).
  • The noise distribution is assumed uniform, with each component sampled from U(0, 1).
  • Differentiable generator and discriminator networks are trained by alternating stochastic-gradient descent steps.

3 CATEGORICAL GENERATIVE ADVERSARIAL NETWORKS (CATGANS)

CatGAN extends adversarial learning to unsupervised and semi-supervised multi-class classification by combining information-maximizing discriminators with generator-based robustness. Its objective makes real-data predictions confident, generated-data predictions uncertain for the discriminator, and class usage balanced.

  • Motivation: Standard GANs cannot directly solve the multi-class task because their binary real-versus-fake features need not align with the desired class structure.The paper motivates CatGAN as a way to regularize classification against spurious correlations and generator-focused features.
  • Framework: CatGAN learns a discriminative classifier for unlabeled data by extending GANs to assign examples to K categories rather than distinguish real from fake.The framework can also be viewed as extending RIM with robustness against a generative adversary.
  • Discriminator requirements: The discriminator seeks confident assignments for data samples, uncertain assignments for generated samples, and equal use of all classes.These requirements are formulated through entropy-based objectives, with the class prior assumed uniform.
  • Generator requirements: The generator seeks samples with certain class assignments while distributing generated samples equally across the K classes.This complements the discriminator's uncertainty about generated samples and supports robustness in the learned classifier.
  • Objective: The CatGAN discriminator objective combines empirical entropy terms into a mutual-information objective over observed data and predicted classes.The discriminator maximizes information about data while minimizing information encoded about generated samples.
  • Implementation considerations: GAN training can be unstable or suffer mode collapse, so the implementation uses batch normalization and hidden-layer noise to stabilize optimization.The paper reports that these measures help prevent generator mode switching and improve discriminator generalization in the few-label setting.

4 EMPIRICAL EVALUATION

CatGAN is evaluated on synthetic clustering and image-recognition tasks, where it generally performs competitively and remains effective with limited or absent labels. The experiments also assess its learned generator qualitatively and quantitatively.

  • 4 EMPIRICAL EVALUATION: Across Tables 1–3, CatGAN is competitive with the state of the art on almost all datasets but is slightly outperformed by the Ladder network with layerwise denoising costs.The evaluation includes classification-error tables for permutation-invariant MNIST, CNNs with reduced labels, and CIFAR-10.
  • 4.1 CLUSTERING WITH CATGANS: CatGAN outperforms k-means and RIM on difficult synthetic clustering examples, while its generator learns to generate the datasets.The comparison includes two-moons and circles data, with the circles results shown in Figure 2.
  • 4.2 UNSUPERVISED AND SEMI-SUPERVISED LEARNING OF IMAGE FEATURES: The image experiments use fully connected and convolutional networks on MNIST and CIFAR-10 with full or reduced labeled sets.Unlabeled examples are retained for semi-supervised or unsupervised learning.
  • 4.2 UNSUPERVISED AND SEMI-SUPERVISED LEARNING OF IMAGE FEATURES: Semi-supervised CatGAN approaches state-of-the-art performance on permutation-invariant MNIST and works remarkably well with only 100 labeled examples.The half-shot procedure achieves 9.7% error without label information during training, while the Ladder network remains better in the reported setting.
  • 4.2 UNSUPERVISED AND SEMI-SUPERVISED LEARNING OF IMAGE FEATURES: Unsupervised CatGAN achieves 4.27 classification error with convolutional networks, while semi-supervised CatGAN matches the state of the art on CIFAR-10 with reduced labels.The reported convolutional experiments use deconvolutional generators on MNIST and CIFAR-10.
  • 4.3 EVALUATION OF THE GENERATIVE MODEL: On MNIST, CatGAN samples achieve a Parzen-window log-likelihood of 237±6 versus 225 ± 2 reported for GANs, although this comparison can be misleading.The paper also shows exemplary generated images for MNIST and CIFAR-10.

5 RELATION TO PRIOR WORK

CatGAN connects to prior unsupervised and semi-supervised learning through both discriminative clustering and generative adversarial modeling. Its framework is presented as related to RIM and compatible with orthogonal GAN extensions.

  • CatGAN builds on literature for unsupervised and semi-supervised category discovery, including entropy-based discriminative clustering.
  • Removing the generator and adding ℓ2 regularization makes the CatGAN objective reduce to regularized information maximization.
  • CatGAN is a direct descendant of GANs, while conditioning and hierarchical generation are described as orthogonal extensions.

6 CONCLUSION

The paper concludes that CatGAN combines a discriminative classifier with an adversarial generator for robust unsupervised and semi-supervised learning. It reports competitive image-classification performance and visually faithful generated images, while relating the objective to GAN and RIM formulations.

  • CatGAN combines neural-network classifiers with an adversarial generative model that regularizes discriminative training.
  • The authors report classification performance competitive with state-of-the-art semi-supervised image-classification results and high visual fidelity from the learned generator.
  • The authors did not find ℓ2 regularization helpful in their experiments.
  • The generator and discriminator objectives connect CatGAN to the GAN framework through a multiclass extension with generated samples driven toward classifier uncertainty.For the binary K = 1 case, generated samples are pushed toward p(y = 1 | G(z), D) = 0.5 rather than 0.
  • CatGAN extends RIM by using generator-sample negative entropy as the discriminator regularizer and training the generator alongside the discriminator.
  • The method assumes a uniform class prior, but the objective can use an arbitrary differentiable prior through KL divergence.

D DETAILED EXPLANATION OF THE TRAINING PROCEDURE

Training alternates discriminator maximization and generator minimization using minibatch approximations to the CatGAN expectations. Noise is handled outside logarithmic entropy terms to avoid numerical instability.

  • Training uses batch size B = 100 and alternates one discriminator gradient-ascent step with one generator gradient-descent step.Each objective expectation is approximated using 100 random examples from the relevant data or generator distributions.
  • Noise is added only to terms outside logarithms because entropy calculations can become unstable for small logarithm inputs.The noisy entropy term is effectively replaced by cross-entropy involving the perturbed network.

D.1.1 SYNTHETIC BENCHMARKS

The synthetic-benchmark setup uses multilayer neural networks with leaky ReLU units and batch normalization, while the image experiments use specified fully connected or convolutional architectures for MNIST and CIFAR-10.

  • Synthetic benchmark architecture: Synthetic benchmarks use three hidden layers of 100 leaky ReLU units for both discriminator and generator networks.Batch normalization and Gaussian noise with standard deviation 0.05 are used, and CatGAN noise vectors have dimensionality 10.
  • Architecture tables: The architecture tables identify the discriminator and generator CNNs used for MNIST and CIFAR-10.
  • MNIST: The MNIST experiment uses fully connected discriminator and generator networks with leaky rectified linearities.The discriminator has five hidden layers with sizes 1000, 500, 250, 250, and 250.
  • Image architectures: MNIST and CIFAR-10 CNN architectures are detailed separately, using components including ReLU, leaky ReLU, fully connected, convolutional, and perforated up-sampling layers.

E.1 QUANTITATIVE EVALUATION OF THE GENERATIVE MODEL

The unsupervised CatGAN model achieves sample log-likelihood comparable to the best existing algorithms, although Parzen-window estimates can make generative-model comparisons misleading.

  • CatGAN performs comparably to the best existing algorithms on this measure.
  • Parzen-window log-likelihood estimates can be misleading for comparing generative models.
  • Table 6 compares different generative models on MNIST using sample log-likelihood.

E.2 ADDITIONAL PLOTS FOR EXPERIMENTS ON SYNTHETIC DATA

Across three synthetic datasets, CatGAN correctly clusters the data and generates points nearly indistinguishable from the observed examples; the plots also show uncertainty and unsupervised label indeterminacy.

  • CatGAN is the only compared model reported to correctly cluster all three synthetic datasets.The comparison includes k-means, RIM, and CatGAN.
  • CatGAN also produces synthetic points that are almost indistinguishable from points in the datasets.
  • In the blobs plot, cyan marks points whose class assignment is close to chance level.The threshold is p(y = k, x, D) < 0.55 for every class k.
  • Because training is unsupervised, class identities are not known beforehand and color assignments can differ across models.

E.3 ADDITIONAL VISUALIZATIONS OF SAMPLES FROM THE GENERATIVE MODEL

Additional visualizations compare clustering behavior on synthetic datasets and display samples generated by CatGAN models trained on MNIST and LFW.

  • The MNIST generator uses the semi-supervised experiment architecture, while the LFW generator uses the CIFAR-10 experiment architecture.
  • The supplementary plots compare k-means, RIM, and CatGAN on the two moons and circles datasets.The two moons comparison uses K = 2; the circles figure complements the main-paper visualization.
  • Figures 7 and 8 show samples generated by CatGAN generators trained on MNIST and cropped LFW images.
Loading 1511.06390v2…