A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis

1 INTRODUCTION

• Text-to-image synthesis translates written descriptions into images with similar semantic meaning, with potential uses in design, editing, games, and pictorial art.The cited applications include computer-aided design, image editing, game engines, and pictorial art generation.Source: §1 · p. 2

1.1 Traditional Learning Based Text-to-image Synthesis

• Early systems matched keywords to likely image parts and optimized their layout, combining search with supervised learning.The process also drew on natural language processing, computer vision, computer graphics, and machine learning.Source: §1.1 · p. 2
• Traditional approaches could alter characteristics of training images but lacked the ability to generate new image content.Attribute2Image instead learned a layered generative model with disentangled latent variables using a variational autoencoder, conditioned on attributes such as gender, hair color, and age (Yan et al., 2016a).Source: §1.1 · p. 2

1.2 GAN Based Text-to-image Synthesis

• GAN-based text-to-image methods combine generative and discriminative learning, with text features and a latent vector serving as generator inputs.The generator attempts to fool the discriminator, which distinguishes synthetic from real samples.Source: §1.2 · pp. 3–4
• The survey organizes GAN-based text-to-image work into semantic, resolution, diversity, and motion enhancement categories.It reviews motivations, representative models, architectures, and possible drawbacks across those groups.Source: §1.2 · p. 4
• The survey also covers related work, GAN concepts, model taxonomies and architectures, benchmarks, metrics, comparisons, and future directions.The review’s framework and methods are visually summarized in Figure 4.Source: §1.2 · pp. 4–5, Figure 4 · p. 4

2 RELATED WORK

• Huang et al. (2018) surveyed GAN image synthesis broadly, including core GAN concepts, image-generation approaches, text-to-image and image-to-image methods, and evaluation metrics.Its text-to-image coverage was less detailed and largely chronological.Source: §2 · p. 5
• Wu et al. (2017) covered both unconstrained and constrained image-synthesis applications, including text-to-image, alongside image and video editing.This survey likewise had a comprehensive scope, whereas the present paper focuses specifically on text-to-image and novel model contributions.Source: §2 · p. 5
• The authors report finding no prior survey focused specifically on text-to-image synthesis using GANs, and describe this paper as the first to do so.They distinguish their work by detailing representative methods, their models, and their contributions.Source: §2 · p. 5

3 PRELIMINARIES AND FRAMEWORKS

• The section frames its GAN background and framework overview as preparation for reviewing text-to-image models and their architectures.It briefly distinguishes simple from advanced frameworks before the next section's objective- and design-based taxonomy.Source: §3 · p. 6, §3.4 · p. 10

3.1 Generative Adversarial Neural Network

• GANs pair a generator that creates samples with a discriminator that distinguishes real from generated data, but simultaneous training can leave neither model converged.The survey describes alternating updates, holding one network's weights fixed while updating the other.Source: §3.1 · pp. 6–7

3.2 cGAN: Conditional GAN

• A cGAN conditions both generator and discriminator on additional information, such as a class label or text, to direct generation toward tailored outputs.The condition must correspond to the real data; mismatched text and samples can teach the model the wrong association.Source: §3.2 · pp. 7–8

3.3 Simple GAN Frameworks for Text-to-Image Synthesis

• For complicated descriptions, label-conditioned cGANs are limited; text-feature methods embed descriptions and concatenate them with other inputs to train the generator and discriminator.The survey identifies GAN-INT-CLS and TAC-GAN as examples of this approach.Source: §3.3 · p. 8

3.4 Advanced GAN Frameworks for Text-to-Image Synthesis

• Advanced frameworks vary their discriminator and generator designs, including multiple discriminators, multistage GANs, progressive training, and hierarchical discriminators.The survey also describes a pyramid generator with three discriminators and a semantic-relevance discriminator as examples of newer designs.Source: §3.4 · pp. 8, 10, Figure 8 · p. 10

4 TEXT-TO-IMAGE SYNTHESIS TAXONOMY AND CATEGORIZATION

• The section proposes a taxonomy to summarize advanced GAN-based text-to-image synthesis frameworks.The taxonomy is intended to organize frameworks by the aspects of synthesis they address.Source: §4 · p. 10
• The taxonomy names Semantic Enhancement, Resolution Enhancement, Diversity Enhancement, and Motion Enhancement GANs as its four categories.Source: §4 · p. 10
• Each category's subsections introduce typical frameworks and their techniques for addressing particular text-to-image synthesis challenges.Source: §4 · p. 10

• Semantic Enhancement GANs target semantic matching by encoding text as dense features and feeding them to an image-generating network.Source: §4.1 · p. 11
• Resolution Enhancement GANs target image quality while preserving text relevance, often passing earlier-stage GAN outputs to later stages.Source: §4.1 · p. 11
• Diversity Enhancement GANs seek varied visual appearances that remain text-related, using an additional component to estimate image-text relevance.Source: §4.1 · p. 11
• Motion Enhancement GANs aim to create temporally coherent image sequences, first generating images for described actions and then aligning them in order.Source: §4.1 · p. 11

4.2 Semantic Enhancement GANs

• Semantic-enhancement work aims to relate generated images to text, though the survey notes that semantic relevance is subjective and images admit diverse interpretations.Source: §4.2 · p. 11

4.2.1 DC-GAN

• DC-GAN is presented as an early GAN for text-to-image synthesis, encoding text features with a character-level hybrid convolutional recurrent network.Its experiments use Caltech-UCSD Birds and Oxford-102 Flowers, with five descriptions per image.Source: §4.2.1 · pp. 11–12

4.2.2 DC-GAN Extensions

• Extensions include GAN-CLS, GAN-INT, and GAN-INT-CLS, which use matching discriminators, text-manifold interpolation, or both to seek semantic matches.Other extensions use perceptual losses and residual transformations to retain source-image features (Dong et al., 2017a; Cha et al., 2017a), while Paired-D GAN uses separate discriminators for foreground and background.Source: §4.2.2 · pp. 12–13

4.2.3 MC-GAN

• MC-GAN combines a source image's background with a text-described foreground object absent from that image.Its synthesis block extracts background features using convolution and batch normalization, while taking foreground features from the preceding layer.Source: §4.2.3 · p. 13
• MC-GAN lets users preserve a base image's background while generating new images.Source: §4.2.3 · p. 13

4.3 Resolution Enhancement GANs

• StackGAN uses two stages for high-resolution generation, StackGAN++ adds multiscale stages and color-consistency regularization, and DM-GAN uses dynamic memory to refine initial images.The survey presents these as approaches to the difficulty of training GANs to generate high-resolution images.Source: §4.3 · p. 13

4.3.1 StackGAN

• StackGAN uses two cascaded GANs: stage I generates a rough 64x64 image, and stage II refines it into a 256x256 image using the text embedding and stage I output.The cascade is described as a sketch-refinement process; earlier stacked approaches cited in the survey did not condition their generators on text.Source: §4.3.1 · pp. 13–14
• Conditioning Augmentation samples conditioning variables from a Gaussian distribution derived from the text embedding, encouraging robustness and more diverse outputs than fixed embeddings.The training image used for comparison remains the same while the conditioning variables vary.Source: §4.3.1 · p. 14

4.3.2 StackGAN++

• StackGAN++ arranges generators and discriminators in a tree-like multistage structure, progressively producing higher-resolution images and combining conditional with unconditional approximation.The first stage uses noise and conditioning variables; later stages reuse the previous image and conditioning variables, with a 256×256 final image by default.Source: §4.3.2 · p. 14

4.3.3 AttnGAN

• AttnGAN adds word-level conditioning and attention that matches words to image regions, allowing consecutive stages to add details region by region.DAMSM measures similarity between the final image and text at sentence and word levels.Source: §4.3.3 · pp. 14–15
• AttnGAN outperforms the other models on IS by a small amount on CUB and greatly on COCO.The comparison covers StackGAN, StackGAN++, AttnGAN, and HDGAN.Source: §4.3.3 · p. 15

4.3.4 HDGAN

• HDGAN nests adversarial objectives in hierarchical networks to regularize mid-level representations and train a single-stream generator with joint discriminators for higher-resolution images.Its visual-semantic similarity measure evaluates image-text consistency, including semantic mapping to areas such as a bird’s wing or a flower’s petal.Source: §4.3.4 · p. 15

4.4 Diversity Enhancement GANs

• The section’s methods target output diversity while retaining semantic relation to text, with approaches moving from class labels toward direct relevance measures and structured inputs.The survey introduces these methods as attempts to maximize output diversity based on text descriptions.Source: §4.4 · p. 15

4.4.1 AC-GAN

• AC-GAN addresses category scalability and output diversity issues by adding an auxiliary classifier that predicts image class labels alongside true-or-fake status.The survey notes that conventional conditional GAN approaches can produce outputs with similar types and visual appearance despite conditioning on text.Source: §4.4.1 · pp. 15–16

4.4.2 TAC-GAN

• TAC-GAN replaces AC-GAN’s class-label conditioning with textual descriptions, while its discriminator predicts image labels and receives text information.On Oxford-102 flowers, its results are reported as slightly better than GAN-INT-CLS and StackGAN.Source: §4.4.2 · p. 16

4.4.3 Text-SeGAN

• Text-SeGAN estimates image-text semantic relevance with a regression layer scored from 0 to 1 instead of predicting class labels, avoiding restriction to certain classes.A higher fractional score represents better semantic relevance.Source: §4.4.3 · p. 16
• Text-SeGAN generates diverse images semantically relevant to input text on Oxford-102 flowers and reports improved inception score over GAN-INT-CLS, StackGAN, TAC-GAN, and HDGAN.The survey reports experiments and validations on the Oxford-102 flower dataset.Source: §4.4.3 · p. 16

4.4.4 MirrorGAN and Scene Graph GAN

• MirrorGAN uses a reverse image-to-text process to validate generated-image consistency, while Scene Graph GAN uses graph convolution, predicted layouts, and cascaded refinement to generate images.Scene Graph GAN’s visual scene graphs let users specify object relationships.Source: §4.4.4 · pp. 16–17

4.5 Motion Enhancement GANs

• This research area extends text-to-image synthesis to video sequences, which the survey identifies as useful for automated assistance or storytelling.Source: §4.5 · p. 17

4.5.1 ObamaNet and T2S

• ObamaNet generates talking-face videos from text through text-to-speech, an audio-synced time-delayed LSTM for mouth shape, and U-Net video generation; T2S maps text to sign-language glosses and skeletal poses before video generation.ObamaNet models only the mouth region, and the survey characterizes its output as video prediction rather than generation because it is not generated from noise.Source: §4.5.1 · p. 17

4.5.2 T2V

• T2V separates static text features for background and layout from dynamic features for actions, using a gist generator and text-to-filter component to generate videos.The reported videos are semantically related to text but low quality, at 64 × 64 resolution.Source: §4.5.2 · pp. 17–18

4.5.3 StoryGAN

• StoryGAN uses sequential generation for multi-sentence stories, with a story encoder for continuity, a context encoder for sequential image generation, and image and story discriminators.The survey describes its evaluations on CLEVR and Pororo cartoon datasets.Source: §4.5.3 · p. 18
• StoryGAN improves generated-video quality on CLEVR and Pororo in SSIM, visual quality, consistency, and relevance, with the last three measures based on human evaluation.Table 1 summarizes GANs and datasets used for validation, with checkmarks indicating evaluation datasets.Source: §4.5.3 · p. 18, Table 1 · p. 18

5.1 Text-to-image Synthesis Applications

• Text-to-image systems aim to translate sequential descriptions into compelling images, with reported applications including faces, birds, and flowers.The survey identifies potential uses in computer-aided design, image editing, and game engines.Source: §5.1 · pp. 18–19

5.2 Text-to-image Synthesis Benchmark Datasets

• CUB and Oxford pair text with individual birds or flowers, whereas COCO contains 328k multi-object images; CIFAR-10 contains 60,000 images across 10 classes.The survey describes these as commonly used evaluation datasets and notes that COCO has 2.5 million labels.Source: §5.2 · p. 19

5.3 Text-to-image Synthesis Benchmark Evaluation Metrics

• IS, FCN scores, and FID assess generated-image quality differently: IS reflects confidence and diversity, FCN scores reflect classification accuracy, and lower FID is better.FID compares generated images with real images from the distribution.Source: §5.3 · p. 19, Table 2 · p. 20

5.4 GAN Based Text-to-image Synthesis Results Comparison

• Inception Scores show only slight text-to-image improvement for StackGAN++ over StackGAN, although its tree-like structure enhances unconditional image generation.The authors suggest that revising generator or discriminator structures can bring moderate text-to-image improvement.Source: §5.4 · p. 20
• DM-GAN has the best performance, followed by Obj-GAN, in the reported comparison; missing data limits evaluation of AttnGAN and HDGAN.The authors compare the latter models visually, judging HDGAN relatively better on CUB and Oxford and AttnGAN more impressive on COCO.Source: §5.4 · p. 20, Table 2 · p. 20

5.5 Notable Mentions

• Beyond text-to-image synthesis, the survey mentions attribute-based face generation, image captioning, and StackGAN++ color regularization for unconditional generation.The color-regularization term aims to keep samples generated from the same input at different stages more consistent in color.Source: §5.5 · p. 21

6 CONCLUSION

• The reviewed methods use GANs, deep convolutional decoder networks, and multimodal learning to generate images from natural-language text.The survey includes methods targeting images beyond the 64×64 swatches generated in earlier work.Source: §6 · p. 22
• The survey groups GAN-based text-to-image frameworks into semantic, resolution, diversity, and motion enhancement categories.The taxonomy organizes methods by their motivations, architectures, differences, evolution, and relationships.Source: §6 · p. 22
• The surveyed methods were evaluated on datasets covering birds, flowers, humans, and other miscellaneous elements.Source: §6 · p. 22
• The authors identify less human intervention and larger output images as future directions for the surveyed learning methods.Source: §6 · p. 22