Source-linked AI summary
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He
TL;DR
Text-to-image GANs lack fine-grained word-level information, limiting detailed synthesis. AttnGAN uses attention-driven, multi-stage refinement and fine-grained image-text matching, outperforming prior state-of-the-art models by 14.14% on CUB and 170.25% on COCO.
Problem
Existing text-to-image GANs condition on a single sentence vector, lacking word-level information needed for fine-grained image synthesis.
Method
AttnGAN combines an attentional generative network for word-conditioned multi-stage refinement with a deep attentional multimodal similarity model for fine-grained matching loss.
Results
14.14% higher on CUB and 170.25% higher on COCO, AttnGAN significantly outperforms previous state-of-the-art GAN models in inception score.
Takeaways & Limitations
Attention visualization demonstrates that AttnGAN automatically attends to relevant words to form conditions for generating different image parts.
Takeaways & Limitations
Applying LDAMSM to images at all generated resolutions did not improve performance and increased computational cost.
Abstract
from arXiv · showhide
In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation. With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different subregions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.
1. Introduction
AttnGAN addresses the loss of word-level information in sentence-conditioned GANs by using attention-driven, multi-stage refinement for fine-grained text-to-image generation. Its attentional generator and DAMSM are presented as novel components, supported by empirical evaluation and detailed analysis.
- Sentence-level conditioning lacks fine-grained word information, limiting image quality, especially for complex scenes such as COCO images.
- AttnGAN introduces attention-driven, multi-stage refinement to synthesize fine-grained images from natural-language descriptions.
- Its attentional generative network generates image subregions by focusing on the words most relevant to each subregion.The model encodes both a global sentence vector and individual word vectors, using the global vector to generate a low-resolution image in the first stage.
- The paper identifies the attentional generative network and DAMSM as AttnGAN’s two novel components.
- The authors report comprehensive empirical evaluation, showing that AttnGAN significantly outperforms previous state-of-the-art GAN models, alongside a detailed analysis.
2. Related Work
Prior text-to-image work progressed from attention-based iterative drawing to conditional and stacked GANs that improved image synthesis. However, attention had not yet been explored in GANs for text-to-image synthesis.
- Attention-based text-to-image models: AlignDRAW extended DRAW to iteratively draw image patches while attending to relevant caption words.This represented an early attention-based approach to text-to-image generation.
- GAN-based text-to-image models: Conditional GANs synthesized plausible images from text, while additional conditions such as object locations enabled better samples.Reed et al. introduced conditional GAN text-to-image synthesis and later incorporated extra conditions.
- GAN-based text-to-image models: Stacked GANs advanced text-to-image synthesis by using different GANs to generate images at different sizes.This work built on GANs’ ability to generate sharper samples.
- Attention mechanisms: Although attention succeeded in captioning, visual question answering, and machine translation, it had not been explored in GANs for text-to-image synthesis.The passage identifies this as an unresolved gap in prior work.
3. Attentional Generative Adversarial Network
AttnGAN combines an attentional generative network with a deep attentional multimodal similarity model to generate images using word-level conditions and fine-grained image-text matching. Its multi-stage generators refine images from small to large scales, while DAMSM supplies the matching loss.
- Architecture: AttnGAN comprises an attentional generative network and a deep attentional multimodal similarity model with complementary generation and matching roles.The generative attention selects word-level conditions for image sub-regions, while DAMSM computes the fine-grained text-image matching loss.
- Attentional generative network: The generative attention model conditions each image sub-region on the words most relevant to that region instead of relying only on a whole-sentence vector.It uses word features and previous-layer image features to compute word-context vectors for image sub-regions before generating the next-stage image.
- Attentional generative network: m generators use hidden states to produce images at small-to-large scales, enabling multi-stage image refinement.The generators are G0 through Gm−1, with corresponding images x̂0 through x̂m−1.
- DAMSM: DAMSM maps image sub-regions and sentence words into a common semantic space to measure word-level image-text similarity and compute a fine-grained generation loss.Its text encoder is a bidirectional LSTM, while its image encoder uses local and global CNN features from Inception-v3.
- DAMSM: LDAMSM is applied only to the final generated image resolution because applying it across all resolutions increased computational cost without improving performance.The authors report that applying LDAMSM to images generated by G0 through Gm−1 did not improve performance.
4. Experiments
Experiments evaluate AttnGAN and its components on CUB and COCO using inception score, R-precision, qualitative attention visualizations, and comparisons with prior GAN models. AttnGAN improves text-image conditioning, resolution, detail refinement, generalization, and state-of-the-art inception scores.
- Evaluation: AttnGAN is evaluated on the CUB and COCO datasets using inception score and complementary R-precision for text-image conditioning.Inception score measures quantitative image quality, while R-precision addresses whether generated images match their text descriptions.
- The DAMSM loss: Increasing λ for LDAMSM substantially improves R-precision on both datasets; on CUB, AttnGAN1 reaches 4.35 inception score and 58.65% R-precision at λ=5.The CUB comparison changes λ from 0.1 to 5, improving inception score from 4.19 and R-precision from 16.55%.
- The attentional generative network: Stacking two attention models raises output resolution from 128×128 to 256×256 and yields higher inception scores on both CUB and COCO.Later stages use word vectors to rectify earlier defects and add details, including fine-grained features such as beaks and eyes.
- Qualitative analysis: Attention visualizations show G0 sketches primitive shapes and colors, while later stages use relevant words and prior image features to add semantic details.The second attention model can recover objects and attributes omitted by the first attention model for higher-resolution generation.
- Generalization ability: Changing highly attended words modifies generated images accordingly, demonstrating sensitivity to input sentences and generation from unseen text descriptions.The experiments use word changes in text descriptions to test output sensitivity and generalization ability.
- Comparison with prior work: 4.36 inception score on CUB surpasses the previous best 3.82, while 25.89 on COCO improves on 9.58 by 170.25%.The comparison uses CUB and COCO test sets against previous state-of-the-art GAN models.
5. Conclusions
AttnGAN is proposed for fine-grained text-to-image synthesis, combining multi-stage attentional generation with a deep attentional multimodal similarity model for fine-grained image-text matching loss.
- AttnGAN is an Attentional Generative Adversarial Network for fine-grained text-to-image synthesis.
- Its novel attentional generative network produces high-quality images through a multi-stage process.
- A deep attentional multimodal similarity model computes the fine-grained image-text matching loss used to train the generator.
- AttnGAN significantly outperforms previous state-of-the-art GAN models.