Source-linked AI summary
Conditional Generative Adversarial Nets
Mehdi Mirza, Simon Osindero
TL;DR
Unconditioned generative models provide no control over generated data modes, while many labeling problems are probabilistic one-to-many mappings. This paper introduces conditional adversarial nets by conditioning both generator and discriminator, demonstrating class-conditioned MNIST generation and multimodal image tagging.
Problem
Unconditioned generative models lack control over generated modes, while image labeling often requires probabilistic one-to-many predictions with multiple appropriate tags.
Method
The paper conditions both generator and discriminator on auxiliary information y, such as class labels or data from another modality.
Results
Conditional adversarial nets generated MNIST digits conditioned on class labels and modeled multimodal tag-vector distributions for automated image tagging.
Takeaways & Limitations
Conditional adversarial nets provide a proof-of-concept for directing generation with labels and producing multiple possible image tags from image features.
Takeaways & Limitations
The conditional results are presented as proof-of-concept, are outperformed by several approaches including non-conditional adversarial nets, and require further architecture and hyperparameter exploration.
Abstract
from arXiv · showhide
Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models. In this work we introduce the conditional version of generative adversarial nets, which can be constructed by simply feeding the data, y, we wish to condition on to both the generator and discriminator. We show that this model can generate MNIST digits conditioned on class labels. We also illustrate how this model could be used to learn a multi-modal model, and provide preliminary examples of an application to image tagging in which we demonstrate how this approach can generate descriptive tags which are not part of training labels.
1 Introduction
This section motivates conditioning adversarial generative models to control generated data modes and introduces experiments on class-conditional MNIST and multimodal MIR Flickr 25,000 learning.
- Background: Adversarial nets avoid Markov chains and inference during learning, use backpropagation for gradients, and incorporate diverse factors and interactions.They were introduced as an alternative framework for training generative models that avoids approximating many intractable probabilistic computations.
- Background: Adversarial nets can produce state of the art log-likelihood estimates and realistic samples.This performance was demonstrated in prior work.
- Motivation: Conditional generation can direct data generation using class labels, partial data for inpainting, or information from another modality.Unconditioned models lack control over which data modes are generated.
- Contributions: The work constructs conditional adversarial nets and evaluates them on MNIST digits conditioned on class labels and MIR Flickr 25,000 for multimodal learning.These experiments demonstrate the proposed model in two settings.
2 Related Work
Related work identifies two challenges: scaling supervised neural networks to many output categories and modeling probabilistic one-to-many mappings. Prior approaches use multimodal representations, conditional probabilistic generative models, Deep Boltzmann Machines, and multimodal neural language models.
- Challenges: Supervised neural networks remain difficult to scale to extremely large numbers of predicted output categories.Existing work also largely learns one-to-one input-output mappings, whereas many problems are probabilistic one-to-many mappings.
- Prior approaches: Multimodal label representations learned from natural language can make prediction errors semantically close to the truth and support generalized predictions.The passage gives predicting “table” instead of “chair” as an example of a semantically close error.
- Prior approaches: Conditional probabilistic generative models address one-to-many mappings by treating the input as a conditioning variable and producing a conditional predictive distribution.This approach instantiates variation in the output through the conditional distribution.
- Related models: A related study trained a multi-modal Deep Boltzmann Machine on the MIR Flickr 25,000 dataset.The passage states that this dataset is also used in the present work.
- Related models: Another study trained a supervised multi-modal neural language model capable of generating descriptive sentences for images.This work demonstrates multimodal generation applied to image descriptions.
3 Conditional Adversarial Nets
Conditional adversarial nets extend generative adversarial nets by conditioning both the generator and discriminator on auxiliary information y. The conditioning is implemented by feeding y as an additional input to both models, enabling information such as class labels or other modalities to guide generation and discrimination.
- Conditional model: Conditional adversarial nets condition both the generator and discriminator on auxiliary information y, such as class labels or data from other modalities.The paper describes this as an extension of generative adversarial nets.
- Conditional model: The conditioning is implemented by feeding y into both the generator and discriminator as an additional input layer.This provides the same auxiliary information to both adversarial components.
- Generator conditioning: In the generator, prior noise p_z(z) and y are combined in a joint hidden representation whose composition remains flexible under adversarial training.The adversarial framework allows considerable flexibility in how the two inputs are composed.
- Discriminator conditioning: In the discriminator, x and y are presented together as inputs to a discriminative function, implemented here as a multilayer perceptron.The conditional objective is formulated as a two-player minimax game.
4 Experimental Results
The experiments demonstrate conditional adversarial generation on MNIST and image tagging, while positioning the reported results as proof-of-concept rather than evidence of superior efficacy. The model generates samples conditioned on class labels or image features, including potentially multi-modal tag distributions.
- MNIST generation: The conditional adversarial net was trained on MNIST images conditioned on class labels encoded as one-hot vectors.Generated samples were organized with each row conditioned on one label and each column representing a different sample.
- MNIST generation: The MNIST conditional adversarial results are comparable to some network-based approaches but are outperformed by several methods, including non-conditional adversarial nets.The authors present these findings as a proof-of-concept and expect further hyper-parameter and architectural exploration could improve performance.
- Image tagging: Conditional adversarial nets were used for automated multi-label image tagging by generating a possibly multi-modal distribution of tag-vectors conditional on image features.Experiments used the MIR Flickr 25,000 dataset, omitted images without tags, and used the first 150,000 examples for training.
- Image tagging: The image-feature convolutional model and language model were kept fixed during adversarial-net training, with backpropagation through them left for future work.Image representations came from a 4096-unit fully connected layer, while word vectors had size 200 and the vocabulary contained 247465 words.
- Image tagging: Evaluation generated 100 samples per image, retrieved the 20 closest vocabulary words for each sample by cosine similarity, and selected the 10 most common words overall.The resulting generated tags were presented alongside user-assigned tags and annotations.
5 Future Work
The paper’s results are extremely preliminary but demonstrate the potential of conditional adversarial nets for useful applications. Future work includes more sophisticated models, deeper analysis, multi-tag set generation, and jointly learning a task-specific language model.
- The results are extremely preliminary, but they demonstrate the potential of conditional adversarial nets for interesting and useful applications.
- Future experiments will develop more sophisticated models and provide a more detailed, thorough analysis of their performance and characteristics.
- Using multiple tags simultaneously as a set-generation problem is expected to improve results over treating each tag individually.The current experiments use each tag individually.
- Another future direction is a joint training scheme that learns a language model suited to the specific task.Prior work has shown that task-specific language models can be learned.