PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Images from Captions with Attention

Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov

arXiv:1511.02793v2cs.LGcs.CV

TL;DR

Existing image generators often lacked conditioning on natural-language descriptions, despite the prevalence of text accompanying images. The paper introduces alignDRAW, which iteratively draws image patches while attending to caption words, and evaluates it on Microsoft COCO. It reports higher-quality samples than other approaches and generalization to novel scene compositions from previously unseen captions.

  • Problem

    Many previous image generative models were unconditioned or conditioned on labels, while real-world images are often accompanied by unstructured textual descriptions.

  • Method

    AlignDRAW extends DRAW with caption representations and soft word attention, iteratively accumulating image patches on a canvas; a Laplacian-pyramid adversarial network sharpens its outputs.

  • Results

    The model generates images corresponding to captions, including novel scenarios unlikely in real life, after learning on Microsoft COCO.

  • Takeaways & Limitations

    Text-conditioned generation enables testing whether the model generalizes to descriptions of scenes not seen during training.

  • Takeaways & Limitations

    The model struggles to represent fine-grained object details and relies on a separate sharpening step because its samples are slightly blurry.

Abstract

from arXiv · show

Motivated by the recent progress in generative models, we introduce a model that generates images from natural language descriptions. The proposed model iteratively draws patches on a canvas, while attending to the relevant words in the description. After training on Microsoft COCO, we compare our model with several baseline generative models on image generation and retrieval tasks. We demonstrate that our model produces higher quality samples than other approaches and generates images with novel scene compositions corresponding to previously unseen captions in the dataset.

1 INTRODUCTION

The paper develops a caption-conditioned image generator that combines sequential drawing with attention to words, enabling image synthesis from descriptions and generalization to unusual, unseen scenes.

  • Motivation: Caption-conditioned image generation addresses a setting where images are commonly accompanied by unstructured textual descriptions.Earlier generative image models were often unconditioned or conditioned only on classification labels.
  • Motivation: Generating realistic images from descriptions combines language modeling and image generation, making it more difficult than caption generation.The paper reverses the usual direction by taking textual descriptions as input and generating relevant images.
  • Approach: The alignDRAW model iteratively draws image patches on a canvas while attending to relevant words in the caption.It extends DRAW with soft word attention and uses a deterministic Laplacian pyramid adversarial network for post-processing.
  • Generalization: The model generates reasonable images for captions describing highly unlikely real-life scenarios, such as a toilet seat open in a grass field.These captions combine common objects with unusual actions or locations.
  • Generalization: The examples include novel compositions such as a stop sign or elephants flying in blue skies and a person skiing on sand.These scenarios are presented as captions describing scenes not seen during training.

2 RELATED WORK

Prior work established several deep generative approaches, including VAEs, DRAW, and Laplacian-pyramid GANs. This paper instead develops a generative model of images conditioned on captions rather than labels or no conditioning.

  • Prior generative models: Earlier image generative models commonly used Boltzmann Machines and Deep Belief Networks, while discriminative deep networks had achieved broader success.The paper situates its work against the comparatively limited success of generative models.
  • VAE and DRAW: Variational Auto-Encoders introduced continuous latent variables, and DRAW extended the VAE framework with differentiable attention.The encoder approximates a posterior distribution and the decoder stochastically reconstructs data from latent variables.
  • Generative adversarial models: GANs use a generator and discriminator, while conditional GANs can be trained at each level of a Laplacian pyramid.The paper later uses a deterministic Laplacian-pyramid adversarial network for image post-processing.
  • Caption conditioning: Unlike many previous approaches conditioned on nothing or on labels, this paper develops a generative model of images conditioned on captions.The distinction establishes caption conditioning as the paper’s central novelty relative to the cited work.

3 MODEL

AlignDRAW extends DRAW into a caption-conditioned image generator that aligns generation steps with caption words, iteratively writes patches to a canvas, and models sequential latent variables. Training uses a variational objective with an inference network, while generation samples from the prior and applies adversarial post-processing to sharpen images.

  • Conditional model: AlignDRAW represents captions as word sequences and images as sequences of patches drawn onto a canvas over time.
  • Language model: A Bidirectional RNN encodes the caption, while an alignment operator produces a dynamic sentence representation for each image-generation step.The dynamic representation is a weighted sum of word representations using alignment probabilities derived from the caption and the generative hidden state.
  • Conditional DRAW network: The generative RNN combines latent samples with the dynamic caption representation, then writes localized Gaussian-filtered patches into a cumulative canvas.The resulting canvas matrix is transformed into a conditional image distribution through a sigmoid; generated images use its conditional mean.
  • Latent variables: The model uses latent variables whose prior mean and variance depend on previous generative hidden states, rather than independent spherical Gaussians.The paper reports improved performance when dependencies between latent variables are included.
  • Learning and generation: Training maximizes a variational lower bound on image likelihood, whereas generation discards the inference network and samples from the prior.A conditional Bernoulli observation model was used because a conditional Gaussian model worked worse in the experiments; adversarial residual processing sharpens generated samples.
  • Inference network: An inference recurrent network reads image patches and combines them with the generative decoder state to form an approximate posterior over latent sequences.Except at the first step, the posterior depends on the image, caption, and previous latent history.

4 EXPERIMENTS

Experiments evaluate caption-conditioned image generation on Microsoft COCO through qualitative edits, attention analysis, generalization tests, and comparisons with other generative models. The model responds to many caption changes and unusual scenarios, while showing limitations in fine-grained object detail and patch-level interpretability.

  • 4.1 MICROSOFT COCO: Microsoft COCO provides 82,783 images with at least five captions each; experiments use the first five captions and resize images to 32×32 pixels.The dataset supports qualitative and quantitative evaluation against related generative models.
  • 4.1.1 ANALYSIS OF GENERATED IMAGES: Changing object colors in captions produces corresponding color changes in generated school buses, despite the training images specifically showing yellow buses.The experiment tests whether the model follows specified attributes rather than reproducing the dominant training color.
  • 4.1.1 ANALYSIS OF GENERATED IMAGES: Changing skies from blue to rainy and grass from dry to green in captions produces the corresponding background changes in generated images.Background edits require alterations over a wider visual area than object-color changes.
  • 4.1.1 ANALYSIS OF GENERATED IMAGES: The model struggles to distinguish visually similar objects such as cats and dogs, revealing difficulty modeling fine-grained object details.This limitation appears when object differences are small in shape and color.
  • 4.1.1 ANALYSIS OF GENERATED IMAGES: The model generates reasonable images for unlikely scenes, such as a toilet seat sitting open in a grass field.These captions test generalization beyond scenarios likely to occur in real life.
  • 4.1.2 ANALYSIS OF ATTENTION: Attention generally concentrates on semantically important words, but generated samples can ignore words such as “sun,” explaining unchanged visual content after caption edits.Flipping “desert” and “forest” preserves similar attention distributions, whereas swapping “beach” and “sun” changes them substantially.
  • 4.1.2 ANALYSIS OF ATTENTION: No significant connection was found between canvas patches drawn at particular time-steps and the words receiving the most attention.This limits interpretation of how attended words correspond to the model’s sequential drawing process.
  • 4.1.3 COMPARISON WITH OTHER MODELS: Variational models achieve higher SSI scores than LAPGAN, while conditional DRAW variants have similar sample quality and LAPGAN outputs appear noisier.Generative-model evaluation combines two metrics with qualitative comparisons because individual metrics have specific drawbacks.

5 DISCUSSION

alignDRAW generates images corresponding to input captions and can generalize to novel scenarios beyond the training set. However, its slightly blurry outputs require a separate sharpening step that is not trained end-to-end.

  • Discussion: alignDRAW generates images corresponding to given input captions and generalizes to novel scenarios beyond the training set.The paper describes examples involving captions for unlikely real-life scenarios.
  • Discussion: The model uses visual attention to decompose image generation into a sequence of drawing steps and word attention to indicate failures.The authors identify these as advantages of the attentional mechanisms.
  • Discussion: Because alignDRAW produces slightly blurry samples, the authors add GAN-generated edges through a separate sharpening post-processing step.The sharpening stage augments alignDRAW samples rather than being integrated into the main model.
  • Discussion: The separate sharpening stage is not ideal because the complete model is not trained end-to-end.The authors identify bypassing this step while producing sharp images directly as future work.

APPENDIX A: MNIST WITH CAPTIONS

The MNIST-with-Captions experiment tests whether alignDRAW can generate digits at caption-specified positions. The model correctly follows digit identities and positions and generalizes to unseen configurations.

  • APPENDIX A: MNIST WITH CAPTIONS: The experiment places one or two MNIST digits on a 60 × 60 blank image and describes their identities and relative positions in artificial captions.Digits occupy corners or non-overlapping horizontal or vertical arrangements.
  • APPENDIX A: MNIST WITH CAPTIONS: The model correctly places specified digits at their described positions and generalizes reasonably well to configurations absent during training.These results are shown with generated images and attention alignments in Figure 6.
  • APPENDIX A: MNIST WITH CAPTIONS: For two-digit captions, attention dynamically shifts to the digit being drawn at each time-step.For single-digit captions, attention remains on the digit word throughout generation.
  • APPENDIX A: MNIST WITH CAPTIONS: Position words receive small attention values in both single- and two-digit generation settings.The passage contrasts this with attention to digit words during generation.

HYPERPARAMETERS

The implementation uses dataset-specific training schedules and language-model configurations, with architectural details summarized in Table 2. The sharpening GAN follows a prior CIFAR edge-residual configuration.

  • HYPERPARAMETERS: alignDRAW parameters use Gaussian initialization with mean 0 and standard deviation 0.01, and RMSprop starts with learning rate 0.001.For COCO, training lasts 18 epochs and the learning rate drops to 0.0001 after 11 epochs; MNIST training lasts 150 epochs with the drop after 110 epochs.
  • HYPERPARAMETERS: Vocabulary sizes are K = 25323 for Microsoft COCO and K = 22 for MNIST with Captions, with uppercase letters converted to lowercase.The language-model hidden-state configuration is specified separately.
  • HYPERPARAMETERS: The language-model hidden states have 128 units, yielding a 256-dimensional concatenated bidirectional-LSTM state.This configuration is part of the language representation used by alignDRAW.
  • HYPERPARAMETERS: The align operator uses l = 512, with v ∈R512, U ∈R512×256, W ∈R512×ngen, and b ∈R512.The architectural configurations are presented in Table 2.
  • HYPERPARAMETERS: The sharpening GAN matches the 28 × 28 CIFAR edge-residual configuration of Denton et al. and is trained for 6 epochs.The cited configuration is used for the sharpening model.

EVALUATION

Evaluation reports variational lower bounds on average train, validation, and test log-probabilities for conditional DRAW models trained on Microsoft COCO. The alignDRAW model does not suffer much from overfitting, but sharpening substantially worsens the results.

  • EVALUATION: Table 3 reports estimated variational lower bounds on average train, validation, and test log-probabilities for conditional DRAW models on Microsoft COCO.The table is the stated quantitative evaluation for this section.
  • EVALUATION: The alignDRAW model does not suffer much from overfitting.This conclusion is drawn from the reported train, validation, and test lower bounds.
  • EVALUATION: The reported results substantially worsen after sharpening test images.Sharpening therefore changes the evaluation outcome for the test images.

APPENDIX C: EFFECT OF SHARPENING IMAGES.

Figure 7 compares generated images before and after sharpening, using an adversarial network conditioned on caption representations.

  • Sharpening transforms the generated images from the top row into the corresponding outputs shown in the bottom row.
  • The adversarial network is trained on residuals of a Laplacian pyramid to sharpen the images.
  • Caption conditioning uses skipthought vectors for the sharpening network.
Loading 1511.02793v2…