Source-linked AI summary
Generative Adversarial Text to Image Synthesis
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee
TL;DR
Generating realistic images directly from natural-language descriptions remains unsolved. This paper introduces a text-conditioned GAN architecture and demonstrates plausible bird and flower synthesis, including improvements on held-out bird categories and broader MS COCO scenes.
Problem
Text-to-image synthesis from single-sentence visual descriptions remains far from solved, despite text’s generality and flexibility for describing visual categories.
Method
The paper trains a DC-GAN conditioned on features from a hybrid character-level convolutional-recurrent text encoder and introduces manifold interpolation regularization.
Results
The model generates plausible images matching bird and flower descriptions, improves synthesis on held-out CUB categories, and generalizes to multi-object MS COCO scenes.
Takeaways & Limitations
The work demonstrates that detailed natural-language descriptions can guide plausible image synthesis across fine-grained categories and more varied scenes.
Takeaways & Limitations
Generated MS COCO scenes are not usually coherent on close inspection, especially when they contain complex multi-object structures.
Abstract
from arXiv · showhide
Automatic synthesis of realistic images from text would be interesting and useful, but current AI systems are still far from this goal. However, in recent years generic and powerful recurrent neural network architectures have been developed to learn discriminative text feature representations. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to generate highly compelling images of specific categories, such as faces, album covers, and room interiors. In this work, we develop a novel deep architecture and GAN formulation to effectively bridge these advances in text and image model- ing, translating visual concepts from characters to pixels. We demonstrate the capability of our model to generate plausible images of birds and flowers from detailed text descriptions.
1. Introduction
The introduction frames text-to-image synthesis as an unsolved task requiring visual-detail text representations and compelling image generation, and presents a conditional GAN approach evaluated on bird and flower descriptions.
- Problem: The work targets direct translation of single-sentence human-written visual descriptions into image pixels, a problem that remains far from solved.Examples describe specific bird and flower characteristics such as beak shape, belly color, petal color, and anther color.
- Motivation: Natural language provides a general, flexible interface for visual categories, whereas attribute representations are discriminative but cumbersome to obtain.The introduction seeks the generality of text descriptions together with the discriminative power of attributes.
- Challenges: Solving the task requires text features that capture important visual details and a synthesis model capable of producing images humans might mistake for real.The authors build on progress in natural-language representation and image synthesis from deep learning.
- Challenges: The conditional distribution of images given text is highly multimodal, with many plausible pixel configurations that can correctly illustrate one description.This multimodality is identified as a difficult issue not solved by deep learning alone.
- Contribution: The authors develop a simple, effective conditional GAN architecture and training strategy for text-to-image synthesis of birds and flowers from human-written descriptions.Evaluation uses Caltech-UCSD Birds and Oxford-102 Flowers, with five collected text descriptions per image, and includes training and held-out categories.
2. Related work
Prior work addressed multimodal representation learning, retrieval, image-conditioned text generation, and convolutional or adversarial image synthesis. This work instead develops a character-level text encoder with a class-conditional GAN for fine-grained text-to-image synthesis, adding a manifold interpolation regularizer.
- Multimodal learning: Multimodal learning has focused on shared representations across modalities and predicting missing data in one modality conditioned on another.Examples include multimodal autoencoders and deep Boltzmann machines jointly modeling audio, video, images, and text tags.
- Image synthesis: Convolutional decoder networks and GANs had generated realistic 3D renderings and compelling high-resolution images, including class-label-conditioned synthesis.Prior systems included deconvolutional networks for chairs and Laplacian-pyramid adversarial generators and discriminators.
- Multimodal learning: The bulk of image-text multimodal work used retrieval, while recent recurrent decoders generated text descriptions conditioned on images.These approaches typically condition a Long Short-Term Memory network on image features.
- Text-to-image synthesis: Mansimov et al. generated images from text captions with a variational recurrent autoencoder and attention, including synthesis of novel text-image combinations.The model painted images in multiple steps and produced reasonable outputs for unusual descriptions such as a flying blue stop sign.
- This work: The proposed approach conditions GANs on text descriptions rather than class labels and provides an end-to-end differentiable character-to-pixel architecture.Its manifold interpolation regularizer significantly improves generated samples, including held-out zero-shot CUB categories; experiments use fine-grained bird and flower datasets with five human-generated captions per image.
3. Background
The background reviews GAN training and the text–image correspondence framework underlying the method. It also describes the hybrid character-level convolutional-recurrent text encoder used for generality and typo robustness.
- Generative adversarial networks: GANs train a generator and discriminator in a two-player minimax game, with the discriminator distinguishing real from synthetic images and the generator fooling it.The game is defined over V(D,G).
- Generative adversarial networks: The GAN minimax game has a global optimum when p_g = p_data, while practical training favors maximizing log(D(G(z))) over minimizing log(1 − D(G(z))).This alternative generator objective addresses the poor samples typically produced at the start of training.
- Text–image correspondence: The text–image model uses deep convolutional and recurrent text encoders to learn a correspondence function, training the text classifier with a structured loss.The training data contain images, corresponding text descriptions, and class labels.
- Text–image correspondence: Image and text encodings are trained so descriptions have higher compatibility with images from their corresponding class than with images from other classes, and vice versa.The image encoder may be a deep convolutional neural network, while the text encoder may be a character-level CNN or LSTM.
- Text encoder: For generality and robustness to typos and large vocabularies, the model always uses a hybrid character-level convolutional-recurrent text network.Its parameters are learned by backpropagating surrogate-objective gradients through the text encoder.
4. Method
The method uses a text-conditioned DC-GAN whose generator and discriminator receive features from a hybrid character-level convolutional-recurrent text encoder. Its GAN-CLS training separates image realism from image-text matching, while embedding interpolation and style inversion extend generation capabilities.
- Architecture: A text-conditioned DC-GAN uses a hybrid character-level convolutional-recurrent encoder, with both generator G and discriminator D performing feed-forward inference conditioned on text features.The generator maps noise and text embeddings to synthetic images, while the discriminator evaluates image-text inputs.
- Generator: The generator compresses the text embedding to 128 dimensions, concatenates it with Gaussian noise, and generates an image through feed-forward deconvolution.The generated image is defined as x̂ ← G(z, ϕ(t)).
- GAN-CLS training: GAN-CLS adds real images paired with mismatched text so the discriminator separately learns image realism and image-text matching.The discriminator scores real matching pairs, real mismatched pairs, and fake images paired with corresponding text.
- Embedding interpolation: Interpolating between training-caption embeddings creates additional synthetic text embeddings without labeling costs, enabling generation between training points on the data manifold.The method uses β = 0.5 in practice, and the interpolated embeddings need not correspond to human-written text.
- Style transfer: A learned convolutional style encoder can invert the generator to recover style factors from images and transfer a query image’s style to a specified text description.The approach treats text embeddings as content and noise as style factors such as background color and pose.
5. Experiments
Experiments on CUB birds and Oxford-102 flowers evaluate GAN, GAN-CLS, GAN-INT, and GAN-INT-CLS in zero-shot text-to-image synthesis. Interpolation-based models produce plausible, caption-matching birds, while all methods generate plausible flowers; further experiments examine style disentanglement, style transfer, and interpolation.
- Datasets and setup: The experiments use class-disjoint CUB and Oxford-102 splits with five captions per image, covering 200 bird and 102 flower categories.CUB contains 11,788 bird images with 150 train+val and 50 test classes; Oxford-102 contains 8,189 flower images with 82 train+val and 20 test classes.
- Zero-shot synthesis: GAN-INT and GAN-INT-CLS generate plausible CUB bird images that usually match all or part of the captions, unlike GAN and GAN-CLS.The interpolation regularizer was needed to reliably achieve visually plausible bird results.
- Zero-shot synthesis: All four methods generate plausible Oxford-102 flower images matching the descriptions, while basic GAN produces more varied flower morphology and other methods produce more class-consistent images.Color information is preserved even when some test-category shapes were unseen during training.
- Style disentanglement: Models with the interpolation regularizer, GAN-INT and GAN-INT-CLS, perform best at style prediction, while captions alone are not informative for style.The evaluation uses pose and background-color verification to assess whether style is separated from image content.
- Style transfer: GAN-INT-CLS transfers pose and detailed background information from an unseen query image onto a text description.Examples include preserving a tree branch on which the bird is perched.
- Interpolation and sample analysis: Text interpolation changes content such as bird color while preserving pose and background, whereas noise interpolation smoothly changes style while keeping content fixed.Sampling multiple noise vectors with one text encoding also produces diverse, sharp samples, although generated scenes are often incoherent on close inspection.
6. Conclusions
The work presents a simple, effective model that generates plausible images from detailed visual descriptions. It improves text-to-image synthesis on CUB and supports disentangling and transfer of visual properties.
- The model synthesizes many plausible visual interpretations of a given text caption.
- The manifold interpolation regularizer substantially improves text-to-image synthesis on CUB.
- The model disentangles style and content and transfers bird pose and background from query images onto text descriptions.