Source-linked AI summary
DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis
Minfeng Zhu, Pingbo Pan, Wei Chen, Yi Yang
TL;DR
Existing text-to-image refinement methods depend on initial-image quality and use unchanged word representations despite differing word importance. DM-GAN introduces dynamic memory with gated text selection and adaptive image–memory fusion; experiments on CUB and COCO report favorable quantitative and qualitative results, including improved CUB IS and FID. The paper notes that final results still rely heavily on the layout of multiple subjects in the initial images.
Problem
Existing multi-stage methods depend heavily on initial-image quality and use unchanged word representations although words differ in importance for depicting image content.
Method
DM-GAN uses dynamic memory to refine fuzzy initial images, a writing gate to select relevant text information, and a response gate to fuse memory and image features.
Results
DM-GAN outperforms previous text-to-image methods quantitatively and qualitatively on CUB and COCO, improving CUB IS from 4.36 to 4.75 and reducing FID from 23.98 to 16.09.
Takeaways & Limitations
DM-GAN can refine initial images with wrong color and rough shapes while generating more photo-realistic images than the compared approaches.
Takeaways & Limitations
Final results still rely heavily on the layout of multiple subjects in the initial images.
Abstract
from arXiv · showhide
In this paper, we focus on generating realistic images from text descriptions. Current methods first generate an initial image with rough shape and color, and then refine the initial image to a high-resolution one. Most existing text-to-image synthesis methods have two main problems. (1) These methods depend heavily on the quality of the initial images. If the initial image is not well initialized, the following processes can hardly refine the image to a satisfactory quality. (2) Each word contributes a different level of importance when depicting different image contents, however, unchanged text representation is used in existing image refinement processes. In this paper, we propose the Dynamic Memory Generative Adversarial Network (DM-GAN) to generate high-quality images. The proposed method introduces a dynamic memory module to refine fuzzy image contents, when the initial images are not well generated. A memory writing gate is designed to select the important text information based on the initial image content, which enables our method to accurately generate images from the text description. We also utilize a response gate to adaptively fuse the information read from the memories and the image features. We evaluate the DM-GAN model on the Caltech-UCSD Birds 200 dataset and the Microsoft Common Objects in Context dataset. Experimental results demonstrate that our DM-GAN model performs favorably against the state-of-the-art approaches.
1. Introduction
Multi-stage text-to-image methods first generate low-resolution images and then refine them, but refinement is limited by poor initial images and unchanged word representations. DM-GAN addresses these issues with dynamic memory, gated text selection, and adaptive image–memory fusion, and reports stronger results on CUB and COCO.
- Multi-stage methods generate low-resolution initial images before refining them into high-resolution images.
- Refinement depends heavily on initial-image quality, while unchanged word representations ignore that words differ in importance for image content.
- DM-GAN adds a dynamic memory mechanism that uses fuzzy initial-image features to retrieve information for image refinement.
- 4.75 versus 4.36 IS and 16.09 versus 23.98 FID are reported for DM-GAN and the comparison method on CUB, respectively; R-precision improves by 4.49% and 3.09% across the two datasets.
- A memory writing gate selects text information relevant to the initial image, and a response gate adaptively fuses memory and image features.
2. Related Work
Related text-to-image work includes single-stage models and multi-stage refinement systems, while memory networks provide explicit storage and attention-based reading. DM-GAN builds on these directions by combining multi-stage synthesis with dynamic key-value memory operations.
- Single-stage: Single-stage methods learn joint image–language embeddings and use generative models to synthesize images from text.
- Multi-stage: StackGAN and StackGAN++ use two stages to generate photo-realistic high-resolution images, while AttnGAN refines images using attention.
- Memory Networks: Memory networks store information explicitly and read memory contents according to relevance, with key-value variants separating weighting keys from retrieved values.
- DM-GAN first generates an initial image and then refines it to produce a high-quality image.
3. DM-GAN
DM-GAN refines rough initial images through dynamic memory operations that retrieve text information relevant to current image features. Gated writing and response mechanisms control which text is stored and how memory outputs are fused during repeated refinement.
- Architecture: DM-GAN uses two stages: an initial generator creates a rough image, followed by dynamic-memory refinement to produce higher-resolution images.The initial stage synthesizes a 64x64 image, while refinement produces 128x128 and 256x256 outputs.
- Dynamic Memory: The refinement stage stores text information in key-value memory, retrieves relevant features using image features, and fuses memory reads with image features.Its four operations are Memory Writing, Key Addressing, Value Reading, and Response.
- Gated Response: A response gate adaptively controls fusion between the information read from memory and the current image features.The fused features are then upsampled and processed by residual blocks to obtain the refined image.
- Gated Memory Writing: A memory writing gate selects words relevant to the current image, rather than writing only partial or unchanged text information.The gate combines image features from the previous stage with word features to calculate word importance before writing memory slots.
- Objective Function: The training objective combines adversarial, conditioning augmentation, and DAMSM losses to encourage realistic images that match their text descriptions.The adversarial objective includes unconditional realism and conditional image-text matching terms, while DAMSM measures image-text matching.
4. Experiments
DM-GAN is evaluated on CUB and COCO using quantitative metrics, qualitative comparisons, refinement stages, diversity tests, and architecture variants. Results show improved image quality, text conditioning, refinement, diversity, and component effectiveness.
- Experimental Setup: Experiments evaluate DM-GAN on CUB and COCO using IS, FID, and R-precision, with 30,000 test-set-conditioned images generated per model.IS measures diversity and class clarity; FID measures distributional distance; R-precision measures text-image conditioning.
- Text-to-Image Quality: DM-GAN improves AttnGAN’s IS from 4.36 to 4.75 on CUB and from 25.89 to 30.49 on COCO.The reported improvements are 8.94% on CUB and 17.77% on COCO.
- Text-to-Image Quality: DM-GAN decreases AttnGAN’s FID from 23.98 to 16.09 on CUB and from 35.49 to 32.64 on COCO.Lower FID indicates a closer generated-image distribution to real images.
- Visual Quality: Qualitative comparisons report more vivid details, clearer backgrounds, and better global structure than AttnGAN, GAN-INT-CLS, and StackGAN.The reported examples include clearer bird-background separation and more coherent multi-subject scenes.
- Visual Quality: Across refinement stages, DM-GAN improves blurry 64×64 initial images into more photo-realistic high-resolution images with clearer backgrounds and finer textures.The refinement process is shown after one and two refinement operations.
- Diversity: Images generated from the same text description are similar but not identical, demonstrating diversity across different noise vectors.Figure 5 compares synthetic images produced from one shared description.
- Ablation Study: Ablation results on CUB show further improvement when the key-value memory, writing gate, and response gate are included.The baseline removes all three components, while variants compare their contributions.
5. Conclusions
DM-GAN combines dynamic memory, a memory writing gate, and a response gate to refine generated images and emphasize important text information. Experiments show strong performance, while multi-subject layouts remain dependent on the initial image.
- DM-GAN uses dynamic memory to refine the initial generated image, a memory writing gate to highlight important text information, and a response gate to fuse image and memory representations.
- Figure 6 compares the top five relevant words selected by attention versus dynamic memory, and by the memory writing step versus key addressing step.
- The conclusion reports that DM-GAN outperforms state-of-the-art methods on two real-world datasets using qualitative and quantitative measures.
- Although DM-GAN can refine initial images with wrong colors and rough shapes, final results still rely heavily on the initial layout of multiple subjects.