PaperlayerView exact revision on arXiv

Source-linked AI summary

Text to Image Generation with Semantic-Spatial Aware GAN

Kai Hu, Wentong Liao, Michael Ying Yang, Bodo Rosenhahn

arXiv:2104.00567v6cs.CVcs.LG

TL;DR

T2I models must generate realistic images whose local regions remain consistent with textual attributes, a challenge existing fusion strategies do not fully solve. SSA-GAN uses semantic-adaptive text-image transformations and weakly supervised spatial masks, and experiments on COCO and CUB report stronger visual fidelity and text alignment than recent methods.

  • Problem

    T2I models can match a description holistically while producing local regions or attributes that are not recognizable or text-consistent.

  • Method

    SSA-GAN uses semantic-adaptive affine transformations conditioned on text and weakly supervised masks based on current text-image fusion to guide spatial refinement.

  • Results

    SSA-GAN outperforms recent state-of-the-art approaches on COCO and CUB in visual fidelity and alignment with input text descriptions.

  • Takeaways & Limitations

    The framework generates images with more vivid details, clearer backgrounds, and semantic consistency with specified attributes and modified text conditions.

Abstract

from arXiv · show

Text-to-image synthesis (T2I) aims to generate photo-realistic images which are semantically consistent with the text descriptions. Existing methods are usually built upon conditional generative adversarial networks (GANs) and initialize an image from noise with sentence embedding, and then refine the features with fine-grained word embedding iteratively. A close inspection of their generated images reveals a major limitation: even though the generated image holistically matches the description, individual image regions or parts of somethings are often not recognizable or consistent with words in the sentence, e.g. "a white crown". To address this problem, we propose a novel framework Semantic-Spatial Aware GAN for synthesizing images from input text. Concretely, we introduce a simple and effective Semantic-Spatial Aware block, which (1) learns semantic-adaptive transformation conditioned on text to effectively fuse text features and image features, and (2) learns a semantic mask in a weakly-supervised way that depends on the current text-image fusion process in order to guide the transformation spatially. Experiments on the challenging COCO and CUB bird datasets demonstrate the advantage of our method over the recent state-of-the-art approaches, regarding both visual fidelity and alignment with input text description.

1. Introduction

T2I is challenging because it requires cross-modal text-to-image transformation while preserving semantic consistency. SSA-GAN addresses limitations in text-image fusion with an end-to-end, one-stage framework using spatially guided semantic transformations.

  • T2I remains difficult because it involves cross-modal transformation and maintaining semantic consistency with the input text.
  • Stacked generator-discriminator systems improve resolution but increase computation and training instability, while early errors can limit later refinement.
  • Existing fusion methods rely on naive concatenation, costly cross-modal attention, or transformations that inadequately connect high-level language semantics with low-level image regions.
  • SSA-GAN uses one generator-discriminator pair and trains end-to-end so its text encoder can learn representations for image generation.
  • Its SSACN block combines semantic-aware affine transformation, a spatial mask predictor, and a residual block to deepen text-image fusion without requiring additional annotations.

2. Related Work

Prior T2I work evolved from stacked GANs and simple text-image fusion toward one-stage generation and more adaptive conditioning. SSA-GAN follows the one-stage structure while replacing its upsampling blocks with SSACN blocks.

  • Stacked T2I GANs generate images from coarse to fine with multiple generator-discriminator pairs, but one-stage models were introduced to avoid their training difficulties.
  • Text-image fusion methods include concatenation, cross-modal attention, dynamic memory refinement, semantic-conditioned normalization, and text-conditioned affine transformations.
  • SSA-GAN adopts a one-stage architecture and replaces UPBlocks with seven SSACN blocks that deepen text-image fusion while improving resolution.

3. Method

SSA-GAN uses a one-stage generator with seven SSACN blocks to fuse text and image features while progressively improving resolution. Its core SSCBN mechanism combines text-conditioned affine transformations with weakly supervised spatial masks that control where text information is reinforced.

  • Architecture: SSA-GAN uses a text encoder, a one-stage generator with seven SSACN blocks, and a discriminator to produce 256 × 256 RGB images from text and noise.The generator deepens text-image fusion and improves resolution across SSACN blocks.
  • Text Encoder: The text encoder is a bidirectional LSTM that produces a 256-dimensional sentence vector and 18 word features, and it can be fine-tuned jointly with the generator.The encoder is initialized from a pretrained model and optimized with the framework in the reported ablation setting.
  • SSACN: Each SSACN block receives text features and image feature maps, uses a residual block to preserve image content, and applies mask-guided text-image fusion.The residual block helps prevent text-irrelevant image parts from being changed or overwhelmed by text information.
  • SSACN: The mask predictor estimates a value in [0, 1] at each spatial location from current image features to determine how strongly later affine transformation should act there.The mask is weakly supervised through the current generated image feature maps and indicates regions needing additional textual reinforcement.
  • Semantic-Spatial CBN: SSCBN learns affine parameters from the text vector and uses predicted masks to control their spatial application, thereby fusing text and image features.The mask determines both where text information is added and how much it is reinforced.
  • Objectives: A one-way discriminator compares generated-image features with the encoded text, while adversarial, MA-GP, and optional DAMSM losses train realism and text-image consistency.The method reports state-of-the-art performance even without DAMSM, according to the supplied passage.

4. Experiments

Experiments on CUB and COCO evaluate SSA-GAN against established text-to-image GANs using IS, FID, qualitative comparisons, mask visualization, and ablations. Results show stronger text-image consistency and visual quality, especially for detailed bird attributes and complex multi-object scenes.

  • Experimental Setup: The study evaluates SSA-GAN on CUB and COCO against StackGAN++, AttnGAN, ControlGAN, SD-GAN, DM-GAN, and DF-GAN using IS and FID.COCO is treated as more challenging because it contains multiple objects and complex backgrounds.
  • Quantitative Results: SSA-GAN improves CUB Inception Score from 4.86 to 5.17 compared with DF-GAN.The paper links higher IS with image quality and text-image semantic consistency.
  • Quantitative Results: SSA-GAN decreases COCO FID from 28.92 to 19.37, while its CUB FID is 15.61 versus 15.30 for StackGAN++.On CUB, its FID is lower than DF-GAN's 19.24 and DM-GAN's 16.09.
  • Qualitative Results: Qualitative CUB comparisons show SSA-GAN preserving detailed attributes such as orange bills, grey crowns, red eyes, and black bills more reliably than competing methods.The comparison also reports more vivid details and clearer backgrounds.
  • Qualitative Results: On COCO, SSA-GAN generates recognizable, separated objects and corresponding backgrounds, whereas competitors produce mixed objects, fuzzy backgrounds, or unnatural scene elements.The examples include clearly separated cows and more natural skier and elephant placements.
  • Qualitative Results: Deeper SSACN stages produce masks that focus first on the whole bird and then on local parts for detail refinement.The mask maps are predicted from current generated image features during progressively deeper text-image fusion.

5. Conclusion

SSA-GAN uses one generator-discriminator pair in an end-to-end T2I framework centered on SSACN. SSACN predicts mask maps from current image features and learns text-conditioned affine parameters to deepen text-image fusion, with experiments and ablations showing improvement over prior state-of-the-art methods.

  • SSA-GAN uses one generator-discriminator pair and is trained end-to-end for text-to-image generation.
  • SSACN predicts mask maps from current generated image features while learning affine parameters from encoded text vectors.
  • The SSACN block deepens text-image fusion throughout image generation and supports text-image consistency.
  • Experiments and ablation studies demonstrate effectiveness and significant improvement over previous state-of-the-art T2I methods.

Appendix

The appendix supplies additional qualitative examples, reviews DAMSM Loss, and provides the implementation code.

  • The appendix provides more qualitative examples for discussion in Section 5.
  • It briefly reviews DAMSM Loss for completeness.
  • The authors make the code available through a GitHub repository.

Qualitative Examples

Qualitative examples indicate that the method generates more realistic images and better matches text-described attributes across bird and complex multi-object scenes. On CUB, it captures details such as an orange bill; on COCO, it handles multiple objects and varied backgrounds.

  • On CUB, the method generates more vivid birds that better match attributes described in the input text.
  • The method generates an orange bill and a more realistic whole bird where competing methods miss the attribute or realism.
  • On COCO, the method generates more realistic complex images containing multiple objects and various backgrounds from text.

Mask Prediction

Predicted mask maps evolve across SSACN stages as text-image fusion deepens. They progress from broad layout and background regions to individual objects and finally object details, refining image features where text information is needed.

  • Early SSACN stages do not clearly indicate where to fuse text information because fusion is shallow and feature maps require more text.
  • From the third stage, mask maps increasingly focus on attributes or objects mentioned in the text descriptions.
  • The fifth, sixth, and seventh stages shift attention from layout and background to individual objects and their details.
  • Mask prediction identifies image-feature regions needing text refinement and deepens text-image fusion during generation.

Diverse Images Generation from Diverse Texts

The method generates diverse images while responding to changes in textual attributes, objects, and backgrounds. Its predicted masks accompany the generated images in these demonstrations.

  • Changing the described color to blue, red, white, or pink produces corresponding images with predicted mask maps.Different noise vectors are sampled for images within the same row.
  • Editing object descriptions changes the generated cattle to brown cattle and then sheep.
  • Editing the background description changes green grass to yellow grass in the generated images.

DAMSM Loss

DAMSM measures image-text correspondence by relating words to image sub-regions and comparing whole images with whole descriptions. It combines word-level and sentence-level matching losses.

  • DAMSM uses semi-supervised matching between entire images and whole sentences at both sentence and word levels.
  • Image encoding provides 768-dimensional local features for 289 image sub-regions and a 2048-dimensional global feature vector.The local features come from Inception-v3’s “mixed 6e” layer, while the global vector comes from its last average-pooling layer.
  • Word-region similarities are normalized to build attention-based region-context vectors for each word.Each context vector is a weighted sum over regional visual vectors, with γ1 = 5 controlling emphasis on relevant sub-regions.
  • Word-level relevance is computed with cosine similarity between each word feature and its image context vector.
  • The loss compares matching image-sentence pairs against mismatching pairs in a batch and combines word-level and sentence-level objectives.For M = 10 paired examples, only the corresponding image and sentence are treated as matching.
Loading 2104.00567v6…