Source-linked AI summary
MirrorGAN: Learning Text-to-image Generation by Redescription
Tingting Qiao, Jing Zhang, Duanqing Xu, Dacheng Tao
TL;DR
Text-to-image generation still struggles to preserve semantic consistency while producing realistic images. MirrorGAN addresses this gap with a text-to-image-to-text framework using semantic embeddings, global-local attentive cascaded generation, and text regeneration; experiments on two benchmark datasets report new state-of-the-art performance. Its design is constrained by incomplete joint end-to-end optimization and basic text embedding and captioning components.
Problem
Text-to-image methods have improved visual realism, but guaranteeing semantic alignment between generated images and input descriptions remains challenging.
Method
MirrorGAN combines STEM word- and sentence-level embeddings, GLAM’s global-local attentive cascaded generators, and STREAM-based text regeneration to learn text-to-image generation by redescription.
Results
MirrorGAN achieves new state-of-the-art performance on two benchmark datasets for visual realism and semantic consistency.
Takeaways & Limitations
The framework uses regenerated descriptions and dual cross-media regulation to align generated images with their text semantics.
Takeaways & Limitations
STREAM and the other MirrorGAN modules are not jointly optimized with complete end-to-end training, and STEM and STREAM use basic embedding and captioning methods.
Abstract
from arXiv · showhide
Generating an image from a given text description has two goals: visual realism and semantic consistency. Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks, guaranteeing semantic consistency between the text description and visual content remains very challenging. In this paper, we address this problem by proposing a novel global-local attentive and semantic-preserving text-to-image-to-text framework called MirrorGAN. MirrorGAN exploits the idea of learning text-to-image generation by redescription and consists of three modules: a semantic text embedding module (STEM), a global-local collaborative attentive module for cascaded image generation (GLAM), and a semantic text regeneration and alignment module (STREAM). STEM generates word- and sentence-level embeddings. GLAM has a cascaded architecture for generating target images from coarse to fine scales, leveraging both local word attention and global sentence attention to progressively enhance the diversity and semantic consistency of the generated images. STREAM seeks to regenerate the text description from the generated image, which semantically aligns with the given text description. Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN over other representative state-of-the-art methods.
1. Introduction
Text-to-image generation must balance visual realism with semantic alignment, but existing GAN-based approaches struggle to guarantee that generated images match their descriptions. MirrorGAN addresses this through a text-to-image-to-text framework combining semantic embeddings, global-local attention, and text regeneration.
- Text-to-image generation seeks visually realistic images that match given text, while semantic alignment remains challenging despite progress from GAN-based methods.
- Existing discriminator-based approaches struggle to model semantic consistency because of the domain gap between text and images.
- MirrorGAN unifies text-to-image and image-to-text generation by using redescription as a semantic constraint on generated images.
- STEM produces word- and sentence-level embeddings, while GLAM progressively generates images from coarse to fine scales using local word and global sentence attention.
- STREAM regenerates text from generated images, and a cross-entropy text-semantics reconstruction loss supervises semantic consistency alongside adversarial losses.
- Experiments on two public benchmark datasets report new state-of-the-art performance for both visual realism and semantic consistency.
2. Related work
Prior work uses bidirectional translation and attention to connect visual and linguistic information, but MirrorGAN applies these ideas specifically to text-to-image generation. Its global-local attention addresses the need for semantically smooth multi-stage image synthesis.
- CycleGAN and DualGAN model bidirectional translation between image domains, providing inspiration for MirrorGAN’s cross-domain design.
- Unlike image-to-image translation methods, MirrorGAN targets the larger cross-media gap between text and images and must handle diverse semantics in both domains.
- Attention mechanisms have been widely used across vision and language tasks, including image captioning, visual question answering, and machine translation.
- Global sentence-level attention complements local word attention in each generator stage to guide semantically smooth, progressively refined synthesis.
3. MirrorGAN for text-to-image generation
MirrorGAN integrates text-to-image generation with image-to-text redescription through STEM, GLAM, and STREAM to preserve semantic alignment. Its cascaded generators use global and local attention, while reconstruction and adversarial objectives train realism and text-image consistency.
- MirrorGAN combines T2I generation with I2T redescription so generated descriptions align semantically with the input text.
- STEM: STEM extracts local word-level and global sentence-level embeddings, with conditioning augmentation producing an augmented sentence vector.
- GLAM: GLAM combines word-level attention with sentence-level attention to use local visual-text interactions and global semantic constraints during generation.
- STREAM: STREAM feeds the final generated image to a pretrained CNN encoder and RNN decoder to regenerate its text description.
- Objective functions: Training alternates generators and discriminators using visual-realism and paired semantic-consistency adversarial losses, plus a CE-based text-semantic reconstruction loss.
4. Experiments
The experiments compare MirrorGAN with state-of-the-art text-to-image methods and evaluate its GLAM and STREAM components through ablation studies.
- Experiments compare MirrorGAN with GAN-INT-CLS, GAWWN, StackGAN, StackGAN++, PPGN, and AttnGAN.
- The evaluation includes ablation studies of GLAM and STREAM.
- The experiments are designed to assess both comparative performance and the contributions of MirrorGAN’s key components.
4.1. Experiment setup
MirrorGAN is evaluated on CUB and MS COCO using Inception Score for image quality and diversity and R-precision for visual-semantic similarity.
- The evaluation uses the CUB bird dataset and MS COCO dataset, both with text descriptions paired with images.
- Datasets: The CUB dataset contains 8,855 training images, 2,933 test images, 200 categories, and 10 descriptions per bird image.
- Metrics: Inception Score measures objectiveness and diversity, while R-precision measures visual-semantic similarity between generated images and text descriptions.
4.2. Main results
MirrorGAN is evaluated against representative text-to-image methods using quantitative scores, visual comparisons, and human perceptual tests. It achieves stronger image quality and text-image semantic consistency than the compared methods, with GLAM improving fine-grained visual details.
- Quantitative results: MirrorGAN achieved the highest Inception Score on both CUB and COCO, improving over AttnGAN from 4.36 to 4.56 and from 25.89 to 26.47, respectively.These results indicate more diverse images of better quality.
- Qualitative results: Compared with AttnGAN, MirrorGAN Baseline generated more details and more consistent colors and shapes, while MirrorGAN further improved fine-grained details and semantic consistency.The baseline uses only word-level attention; MirrorGAN additionally uses GLAM's global and local attention collaboratively.
- Qualitative results: In a human perceptual test with 100 volunteers, MirrorGAN images were preferred over AttnGAN images for both authenticity and semantic consistency.The semantic consistency test compared generated images with the ground-truth image under the same text description.
4.3. Ablation studies
Ablation studies examine the effects of the reconstruction-loss weight, STREAM, and GLAM, while visualizations show how global and local attention guide cascaded generation. Larger reconstruction-loss weights and the proposed modules improve reported quality and semantic alignment.
- Ablation studies: Increasing λ from 5 to 20 raised CUB Inception Score from 4.01 to 4.54 and R-precision from 32.07% to 57.67%.On COCO, the corresponding increases were 21.85 to 26.21 and 52.55% to 74.52%; λ = 20 was used by default.
- Ablation studies: Integrating STREAM increased Inception Score from 3.91 to 4.47 on CUB and from 19.01 to 25.99 on COCO, with R-precision showing the same trend.The authors attribute this to stricter semantic alignment through sequential redescription of generated images as the input text.
- Ablation studies: Adding GLAM after STREAM further improved Inception Score and R-precision to achieve new state-of-the-art performance.GLAM combines global and local attention to support visually realistic and semantically consistent generation.
- Visual inspection: Across cascaded stages, images progress from coarse 64 × 64 outputs to finer 128 × 128 and 256 × 256 outputs as attention focuses on increasingly relevant regions and words.Global attention shifts from broad context to specific regions, while local attention guides fine-grained detail synthesis.
- Visual inspection: Modifying a single word in the text descriptions produced images reflecting subtle semantic differences, with corresponding top-2 attention maps shown in the final stage.This visualization probes the generator's sensitivity to localized textual changes.
4.4. Limitation and discussion
The authors identify limitations in MirrorGAN’s training and component choices, while suggesting future exploitation of its complementarity with CycleGAN.
- STREAM and the other MirrorGAN modules are not jointly optimized with complete end-to-end training because of limited computational resources.
- STEM uses a basic text-embedding method, and STREAM uses basic image captioning that could be improved with newer models such as BERT.
- Although designed for cross-media semantic alignment in T2I, MirrorGAN’s complementarity with CycleGAN could be further exploited to enhance joint cross-media modeling capacity.
5. Conclusions
The conclusion presents MirrorGAN as a text-to-image-to-text framework that learns generation by redescription and achieves new state-of-the-art performance on two benchmark datasets.
- MirrorGAN addresses text-to-image generation with a global-local attentive and semantic-preserving text-to-image-to-text framework.
- Its STEM, GLAM, and STREAM modules provide word- and sentence-level embeddings, coarse-to-fine attentive image generation, and text regeneration for semantic alignment.
- MirrorGAN achieves new state-of-the-art performance on two benchmark datasets.
- The work is supported by Chinese, Australian, and national research funding programs.