PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Contrastive Learning for Text-to-Image Generation

Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang

arXiv:2101.04702v5cs.CV

TL;DR

Text-to-image synthesis seeks realistic, coherent images that faithfully match descriptions despite the differing statistics of language and vision. XMC-GAN uses a one-stage GAN with inter- and intra-modal contrastive losses, an attentional self-modulation generator, and a contrastive discriminator. It improves prior results across challenging datasets, including an MS-COCO FID reduction from 24.70 to 9.33 and strong human preferences for realism and alignment.

  • Problem

    Text-to-image systems must produce realistic scenes with high semantic fidelity while learning from unstructured descriptions and differing language–vision statistics.

  • Method

    XMC-GAN is a one-stage GAN that maximizes mutual information with inter- and intra-modal contrastive losses, using attentional self-modulation and a contrastive discriminator.

  • Results

    XMC-GAN establishes marked improvements over previous models across multiple datasets, including reducing MS-COCO FID from 24.70 to 9.33 and receiving 77.3% realism preference.

  • Takeaways & Limitations

    The results support a simpler, end-to-end approach that generates higher-quality images matching both short descriptions and long, detailed narratives.

Abstract

from arXiv · show

The output of text-to-image synthesis systems should be coherent, clear, photo-realistic scenes with high semantic fidelity to their conditioned text descriptions. Our Cross-Modal Contrastive Generative Adversarial Network (XMC-GAN) addresses this challenge by maximizing the mutual information between image and text. It does this via multiple contrastive losses which capture inter-modality and intra-modality correspondences. XMC-GAN uses an attentional self-modulation generator, which enforces strong text-image correspondence, and a contrastive discriminator, which acts as a critic as well as a feature encoder for contrastive learning. The quality of XMC-GAN's output is a major step up from previous models, as we show on three challenging datasets. On MS-COCO, not only does XMC-GAN improve state-of-the-art FID from 24.70 to 9.33, but--more importantly--people prefer XMC-GAN by 77.3 for image quality and 74.1 for image-text alignment, compared to three other recent models. XMC-GAN also generalizes to the challenging Localized Narratives dataset (which has longer, more detailed descriptions), improving state-of-the-art FID from 48.70 to 14.12. Lastly, we train and evaluate XMC-GAN on the challenging Open Images data, establishing a strong benchmark FID score of 26.91.

1. Introduction

Text-to-image synthesis must handle unstructured descriptions and differing language–vision statistics while producing realistic, coherent, and semantically aligned images. XMC-GAN addresses this with a one-stage GAN using multiple contrastive losses and reports strong quality and alignment improvements.

  • Text-to-image synthesis must learn from unstructured descriptions while handling different statistical properties between vision and language inputs.
  • XMC-GAN is a simple one-stage GAN that does not require object-level annotation and can outperform prior object-driven and multi-stage approaches.
  • XMC-GAN maximizes mutual information through image-sentence, region-word, and image-image contrastive losses.
  • FID decreases from 24.70 to 9.33 on MS-COCO and from 48.70 to 14.12 on LN-COCO.
  • Human evaluators rank XMC-GAN best in 77.3% of realism comparisons and 74.1% of text-alignment comparisons.
  • XMC-GAN produces more coherent, detailed, and realistic images that better capture named objects and background compositions.

2. Related Work

Prior text-to-image systems use conditional GANs, multi-stage refinement, or hierarchical object modeling, often requiring additional structure or annotations. XMC-GAN instead applies inter- and intra-modal contrastive learning in text-to-image synthesis.

  • GAN-based text-to-image systems include conditional, coarse-to-fine, and multi-stage approaches for improving sample quality and detail.
  • Hierarchical object-driven methods explicitly model objects but require fine-grained labels such as bounding boxes or segmentation maps.
  • Multi-stage and multi-step frameworks use multiple generators or discriminators, whereas XMC-GAN uses a single generator and discriminator trained end-to-end.
  • Contrastive learning has been used for augmentation consistency, class-conditional generation, disentanglement, and image-to-image translation.
  • XMC-GAN combines intra-modality image-image and inter-modality image-sentence and region-word contrastive learning for text-to-image synthesis.

3. Preliminaries

Contrastive learning learns representations by comparing views through mutual dependence, while GANs combine a generator and discriminator for conditional image synthesis. These preliminaries motivate contrastive objectives alongside adversarial training.

  • 3.1. Contrastive Representation Learning: Contrastive learning learns useful features from different data views by measuring their mutual dependence.
  • 3.1. Contrastive Representation Learning: InfoNCE maximizes a lower bound on mutual information by scoring a matching positive pair above M−1 negative samples.
  • 3.1. Contrastive Representation Learning: The contrastive score function typically uses two parameterized feature encoders, which may share parameters when both views come from the same modality.
  • 3.2. Generative Adversarial Networks (GANs): A GAN uses a generator to map latent noise z to synthesized data and a discriminator to distinguish synthesized inputs from real data.
  • 3.2. Generative Adversarial Networks (GANs): The hinge loss is a commonly used adversarial objective in state-of-the-art GANs for image generation.
  • 3.2. Generative Adversarial Networks (GANs): In conditional GANs, both generator and discriminator receive a condition c, so generated samples should be realistic and match that condition.

4. Method

XMC-GAN maximizes mutual information through contrastive losses spanning image-sentence, image-image, and region-word correspondences. Its one-stage attentional self-modulation generator and dual-role discriminator jointly support text-aligned image synthesis and contrastive learning.

  • Contrastive Losses: XMC-GAN maximizes mutual information by optimizing contrastive losses between image-sentence, generated-real image, and image-region–word pairs.The losses optimize a lower bound on mutual information using InfoNCE objectives.
  • Contrastive Losses: Image-sentence contrastive learning maps global image and sentence representations into a shared joint embedding space.The image and sentence encoders use cosine similarity with a temperature parameter.
  • Contrastive Losses: Region-word contrastive learning uses attention to connect image regions with words without requiring fine-grained word-region annotations.Pairwise cosine similarities produce soft attention weights for aligning words and regions.
  • Attentional Self-Modulation Generator: The one-stage generator directly produces images at the desired resolution from Gaussian noise and BERT-derived sentence embeddings.Global conditioning controls conditional batch normalization through self-modulation.
  • Attentional Self-Modulation Generator: Attentional self-modulation adds word-context vectors as region-specific modulation parameters to generate fine-grained recognizable regions.The mechanism supplements global sentence conditioning with word-level information for each sub-region.
  • Contrastive Discriminator: The discriminator serves both as a real-versus-fake critic and as an encoder for global image and region features used by contrastive losses.Down-sampling blocks produce region features and global image features before separate projection heads.

5. Evaluation

The evaluation covers three challenging datasets using automated image-quality and text-image-alignment metrics alongside human judgments. It includes longer, more descriptive narratives and complex Open Images data, with validation results generated from 30,000 random captions.

  • Datasets: XMC-GAN is comprehensively evaluated on MS-COCO, Localized Narratives COCO, and Localized Narratives OpenImages.LN-COCO narratives are four times longer than MS-COCO captions on average, while LN-OpenImages contains 8.4 objects per image on average.
  • Evaluation Protocol: 30,000 random validation captions are used to generate images for evaluation, with oversampling when a validation set is smaller.The evaluation uses several complementary measures.
  • Automated Metrics: Image quality is assessed with Inception Score and Fréchet Inception Distance, while text-image alignment uses R-precision and semantic object accuracy.SOA measures whether desired objects are detected in generated images, complementing caption-level retrieval metrics.
  • Human Evaluation: Human evaluation ranks anonymized, randomly ordered outputs from four models for realism and language alignment.For 1,000 captions, five independent annotators rank generated images from best to worst on each criterion.

6. Experiments

XMC-GAN achieves strong image quality and text alignment across COCO-14, LN-COCO, and LN-OpenImages, while ablations show complementary contrastive losses are central to its performance.

  • COCO-14: 77.3% of realism comparisons and 74.1% of text-alignment comparisons ranked XMC-GAN first against three recent models on COCO-14.XMC-GAN was also visually judged to produce clearer objects and more coherent scenes than competing models.
  • COCO-14: FID improved from 24.70 to 9.33 on COCO-14, while R-precision reached 71% versus 59% for other models.The independently trained encoders yielded improved fidelity to conditioning captions.
  • LN-COCO: FID improved by 34.58 and IS by 7.07 on LN-COCO compared with TReCS, while R-precision improved by 23.04% over AttnGAN.XMC-GAN produced clearer and more coherent outputs despite LN-COCO’s longer descriptions.
  • LN-OpenImages: On LN-OpenImages, XMC-GAN achieved IS 24.90, FID 26.91, and R-precision 57.55, establishing a benchmark on this challenging dataset.The dataset has greater image and description diversity than MS-COCO.
  • Ablations: Individual contrastive losses improved all metrics and training stability, with image-sentence and region-word losses reducing FID from 39.28 to 19.25 and 24.38.Intermodal losses produced larger FID gains than the discriminator-feature image-image loss, which reached 29.71.
  • Ablations: Combining image-sentence and region-word losses achieved FID 14.25, outperforming either alone, while the best FID used both intermodal losses plus image-image VGG loss.The ablations identify complementary global, local, and intra-modal contrastive signals.

7. Conclusion

The work presents an end-to-end cross-modal contrastive learning framework for GAN-based text-to-image synthesis. Across multiple datasets and evaluations, XMC-GAN generates higher-quality images that better match descriptions while remaining simpler than previous models.

  • XMC-GAN uses cross-modal contrastive learning to train GAN models for text-to-image synthesis.
  • XMC-GAN improves image quality and correspondence to input descriptions across human and automated evaluations on multiple datasets.
  • XMC-GAN performs well on long, detailed narratives while using a simpler, end-to-end model.

A. Implementation Details

The implementation uses standard TensorFlow 2.0 training with spectral normalization, Adam optimization, and fixed generator–discriminator update settings. Models are trained with large batches for 1000 epochs, and reported checkpoints are selected by validation FID.

  • All models use TensorFlow 2.0, with spectral normalization in every discriminator convolutional and fully connected layer.
  • Adam uses β1 = 0.5 and β2 = 0.999, with learning rates of 1e−4 for the generator and 4e−4 for the discriminator.
  • The discriminator is trained for two steps per generator step, and validation uses generators with exponential moving averaged weights.
  • 1000 epochs and batch size 256 are used for reported results, selecting the checkpoint with the best validation FID.
  • Main results use base channel dimension ch = 96, while main-paper ablations use ch = 64.

B. Architecture Details

The appendix points to detailed generator and discriminator architectures and illustrates their sampling blocks. It also describes the interface used to collect human rankings of generated images.

  • Architecture Details: Detailed generator and discriminator architectures are provided in Tables 8a and 8b.
  • Architecture Details: Figure 6 shows the up-sampling and down-sampling blocks used in the architectures.
  • Human Evaluation Interface: Human evaluators rank four randomly ordered images for image realism and alignment to a given caption.

D. Similarities and differences between DAMSM and the proposed contrastive losses

The proposed contrastive losses differ from DAMSM in which networks they train, their mutual-information motivation, and their end-to-end training procedure. The accompanying experiments also examine diversity, caption sensitivity, and performance boundaries across datasets and metrics.

  • Differences from DAMSM: XMC-GAN applies contrastive losses to both the generator and discriminator, whereas DAMSM trains only the generator.
  • Differences from DAMSM: Training discriminator features for contrastive losses makes them more robust and discriminative, and the model is less prone to mode collapse.
  • Differences from DAMSM: XMC-GAN maximizes mutual information through contrastive InfoICE losses using cosine distance, while DAMSM is motivated by information retrieval and sometimes uses dot products.
  • Differences from DAMSM: XMC-GAN trains completely end-to-end, whereas AttnGAN requires separate DAMSM pretraining before the main generator and discriminator models.
  • Evaluation: XMC-GAN improves state-of-the-art FID and human preference, although CP-GAN has better IS and SOA scores.
  • Noise and Diversity: XMC-GAN generates diverse images from one caption across varying noise vectors, unlike CP-GAN outputs that show little diversity.
  • Caption Effects: More detailed LN-COCO captions increase generation difficulty and appear to produce less coherent scenes than MS-COCO captions.

E.3. Random samples

Random qualitative samples show XMC-GAN producing clearer, more coherent scenes on COCO-14 and clearer, more distinct objects and people on LN-COCO. Additional figures compare generated images across captions, noise vectors, and datasets.

  • COCO-14: COCO-14 samples appear significantly clearer and more coherent, often depicting distinct objects compared with previous methods.The examples are randomly selected captions shown in Figure 10.
  • LN-COCO: LN-COCO samples show clear improvements over previous methods despite the greater challenge posed by longer captions.Generated objects and people appear clearer and more distinct in most images.
  • LN-OpenImages: LN-OpenImages examples display original images alongside XMC-GAN generations for a previously untested dataset with greater image complexity and diversity.These comparisons are presented in Figure 12.
  • Qualitative comparisons: Figure 8 compares CP-GAN and XMC-GAN generations for the same caption using different noise vectors.The comparison focuses on how outputs vary under noise changes while holding the caption fixed.
  • Qualitative comparisons: Figure 9 compares generated images for varying captions from COCO-14 and LN-COCO that correspond to the same original image.This setup illustrates qualitative variation under different textual descriptions.
Loading 2101.04702v5…