PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Text-to-Image Synthesis: A Review

Stanislav Frolov, Tobias Hinz, Federico Raue, Jörn Hees, Andreas Dengel

arXiv:2101.09983v2cs.CV

TL;DR

Text-to-image synthesis has advanced in realism, diversity, and semantic alignment, but complex multi-object generation and reliable human-correlated evaluation remain difficult. This review contextualizes the field, proposes a supervision-based taxonomy, examines evaluation methods, and identifies research directions.

  • Problem

    Text-to-image synthesis seeks to connect language with visual understanding and generate images reflecting textual descriptions, but complex scenes and reliable evaluation remain challenging.

  • Method

    The review surveys state-of-the-art adversarial text-to-image models, organizes them by supervision level, and critically examines datasets, architectures, training, and evaluation techniques.

  • Results

    The review finds substantial progress in caption-based synthesis, while complex scenes with multiple interacting objects remain very difficult; best image quality uses semantic masks and process decomposition.

  • Takeaways & Limitations

    Future progress should target better datasets and metrics, improved scene understanding, fine-grained control, and iterative or interactive image manipulation.

  • Takeaways & Limitations

    Automatic metrics remain proxies for human judgment, requiring user-study verification, especially for image-text alignment and subtle numerical or positional information.

Abstract

from arXiv · show

With the advent of generative adversarial networks, synthesizing images from textual descriptions has recently become an active research area. It is a flexible and intuitive way for conditional image generation with significant progress in the last years regarding visual realism, diversity, and semantic alignment. However, the field still faces several challenges that require further research efforts such as enabling the generation of high-resolution images with multiple objects, and developing suitable and reliable evaluation metrics that correlate with human judgement. In this review, we contextualize the state of the art of adversarial text-to-image synthesis models, their development since their inception five years ago, and propose a taxonomy based on the level of supervision. We critically examine current strategies to evaluate text-to-image synthesis models, highlight shortcomings, and identify new areas of research, ranging from the development of better datasets and evaluation metrics to possible improvements in architectural design and model training. This review complements previous surveys on generative adversarial networks with a focus on text-to-image synthesis which we believe will help researchers to further advance the field.

1. Introduction

Text-to-image synthesis aims to create images that reflect textual meaning, motivated by human visual imagination and enabled by conditional generative models. This review contextualizes the field’s development, taxonomy, evaluation practices, and open challenges.

  • Motivation and scope: GANs formulate image synthesis as a competition between a generator producing realistic samples and a discriminator distinguishing real from generated images.This adversarial framework enabled later conditional image-generation approaches.
  • Motivation and scope: Text-to-image synthesis seeks to generate images that correctly reflect textual descriptions and can represent objects, attributes, spatial arrangements, relationships, and detailed scenes.The review frames T2I as an inverse task to image captioning, with text supplying richer semantic information than labels.
  • Field development: Since its 2016 inception, T2I research has improved generated-image quality, dataset complexity, and image resolution, progressing from restricted 64 × 64 datasets to more challenging benchmarks.Early work used Oxford-102 Flowers and CUB-200 Birds; later research incorporated datasets such as COCO.
  • Open challenges: Current models still struggle with complex multi-object scenes, resolutions above 256 × 256 pixels, reproducible quantitative results, and standardized human evaluation.The review identifies these as continuing field challenges rather than resolved capabilities.
  • Review contributions: The review organizes T2I methods by supervision level, distinguishes single-caption approaches from methods using additional information, and critically examines evaluation metrics and future research directions.Its proposed research areas include better datasets and metrics, architectural design, and model training.

2. Fundamentals

The review introduces GAN, conditional GAN, text-encoding, and dataset fundamentals underlying T2I methods. It emphasizes conditioning mechanisms, adversarial training, textual representations, and the differing complexity of standard datasets.

  • Generative adversarial networks: An unconditional GAN maps randomly sampled noise to images, while its discriminator distinguishes real data from generated samples during adversarial training.The generator is trained to capture the real data distribution and fool the discriminator.
  • Conditional GANs: A cGAN adds a conditioning variable to both generator and discriminator networks so the generated image can be controlled by information such as a class label.In the described example, noise and the label jointly form the generator’s hidden representation.
  • Conditional GANs: AC-GAN extends conditional GAN training with an auxiliary classification loss applied to the discriminator.This additional loss is denoted LC in the architecture description.
  • Text encoding: Text encoders convert descriptions into conditioning embeddings, but designing embeddings useful for image generation is nontrivial.Reviewed approaches include char-CNN-RNN, Conditioning Augmentation, BiLSTM with word-level alignment, and transformer-based encoders.
  • Datasets: Oxford-102 Flowers and CUB-200 Birds contain around 10k single-object images with ten captions each, whereas COCO has around 123k images with five captions and usually complex multi-object scenes.This dataset contrast reflects the increasing complexity of T2I evaluation settings.

3. Direct T2I Methods

Direct T2I methods condition GAN-based image synthesis on text and progressively improve resolution, structure, and text-image alignment through stacked, attention-based, memory, and adapted architectures.

  • First T2I Approaches: Early T2I models replace class labels with sentence embeddings and train discriminators to distinguish matching from mismatching image-text pairs.GAN-INT-CLS generated 64 × 64 images, while TAC-GAN added an auxiliary classification loss and generated 128 × 128 images.
  • Stacked Architectures: StackGAN uses a coarse-to-fine two-stage pipeline that transforms a 64 × 64 image into a 256 × 256 image using textual conditioning at both stages.Each discriminator distinguishes matching from nonmatching image-text pairs.
  • Stacked Architectures: StackGAN++ jointly trains three generators and discriminators end-to-end across multiple scales, using Gaussian-sampled text embeddings and color-consistency regularization.The architecture simultaneously approximates multi-scale conditional and unconditional image distributions.
  • Stacked Architectures: Single-generator variants use multi-scale discriminators or feature fusion to increase resolution while combining semantically strong low-resolution features with high-resolution details.HDGAN targets 512×512 images with hierarchically nested discriminators, while PPAN and HfGAN combine multi-scale features using pyramid or hierarchical fusion designs.
  • Attention Mechanisms: AttnGAN adds word-region attention and a DAMSM similarity loss so generation can focus on relevant words when synthesizing fine-grained image details.The model uses sentence-level and word-level information to align local image regions with input text.
  • Attention Mechanisms: Other direct methods refine conditioning through object-grid phrase attention, keyword-focused attention competition, channel-wise attention, Siamese contrastive learning, and semantic-conditioned normalization.These designs target more selective visual-word alignment and adaptation of visual feature maps to linguistic cues.

4. T2I Methods with Additional Supervision

Methods with additional supervision augment captions with multiple descriptions, dialogue, layouts, masks, or scene graphs to provide more structure for text-to-image generation and control.

  • Overview: Additional supervision can improve state-of-the-art performance, but it requires extra annotations such as multiple captions, dialogue, layouts, scene graphs, or masks.The review organizes these methods by the additional inputs used during training.
  • Multiple Captions: Multiple-caption methods improve scene description by enforcing cross-caption consistency or retrieving compatible caption-image items to guide generation.C4Synth iterates over captions to distill concepts, whereas RiFeGAN enriches descriptions through caption matching without an image-captioning network.
  • Dialog: Dialogue-based methods address scenes with interacting objects by adding question-answer information, and ChatPainter reported better performance with a recurrent encoder than a non-recurrent encoder.ChatPainter combines Visual Dialog turns with COCO captions.
  • Dialog: VQA-based methods condition generation on locally related question-answer text, with VQA-GAN additionally requiring layout supervision while another approach improves quality and alignment without changing the architecture.The architecture-preserving method concatenates QA pairs as training samples and adds an external VQA loss.
  • Layout: Layout supervision represents objects with bounding boxes and class labels, providing more structure, better localization, user-controlled generation, and automatic image annotations.Text-layout models use object locations, keypoints, masks, or object pathways to guide image synthesis.
  • Masks and Scene Graphs: Mask-based methods provide object-shape information, while scene-graph methods represent object relationships and convert predicted boxes and masks into layouts for cascaded image generation.Sparse instance masks support editing operations such as adding or removing objects, and scene graphs encode relations including left of, right of, above, below, inside, and surrounding.

5. Evaluation of T2I Models

Evaluating text-to-image models is difficult because image quality, diversity, and image-text alignment require different measures, while current metrics and user studies have important reliability and comparability problems.

  • Evaluation goals: Evaluation must assess both generated-image quality and semantic alignment with the input text.Visual realism and diversity are distinct from whether images correctly reflect descriptions.
  • Image quality metrics: The Inception Score uses classifier confidence and marginal diversity, but can reward memorization or one perfect image per class.It also ignores ground-truth images and relies on an ImageNet classifier, limiting suitability for complex multi-object scenes.
  • Image quality metrics: FID compares real and generated feature distributions, yet assumes Gaussian features, has a biased estimator, and inherits ImageNet-classifier limitations.KID removes estimator bias but can have high variance when per-class sample counts are low.
  • Metric shortcomings: Current models can match real COCO images on IS, R-precision, and CIDEr despite remaining very unrealistic, showing that some metrics may be unreliable.Reported FID, VS similarity, and SOA remain worse than real-image scores, while SOA may be more informative than VS because VS has high standard deviation.
  • Metric shortcomings: IS and FID are poorly suited to complex COCO scenes because ImageNet pretraining and single-object assumptions can misrepresent multi-object content.Object-crop evaluation, including SceneFID, is proposed as a possible mitigation.
  • Reproducibility: Reported scores for identical models vary substantially, and inconsistent evaluation procedures can undermine fair comparisons and reproducibility.The review recommends precise evaluation descriptions, explanations of discrepancies, and released evaluation code.
  • Alignment evaluation: Image-text alignment remains difficult to define, and existing proxies do not cover many desired aspects or consistently correlate with human judgments.User studies are also difficult to compare because setups differ, and they are expensive and time consuming.
  • Overall assessment: The review concludes that evaluation remains difficult and that numerous recent approaches may create false confidence about model performance.It identifies fair, standardized user studies as the current true way to evaluate performance.

6. Discussion & Challenges

The review summarizes progress in adversarial T2I synthesis while identifying persistent difficulties in complex scenes, text representations, evaluation, datasets, and alternative generative models.

  • 6. Discussion & Challenges: Current T2I systems have progressed from simple architectures to multi-stage pipelines with multiple losses and more realistic outputs.Progress is strongest for flowers, birds, and high-resolution faces, while COCO images and individual objects still lack fine-grained detail and sharpness.
  • 6. Discussion & Challenges: Generating complex scenes with multiple interacting objects remains difficult compared with high-quality single-object synthesis.The review connects this difficulty to scene complexity and limitations in available textual descriptions.
  • 6. Discussion & Challenges: The influence of linguistic properties and text-embedding quality on final T2I performance remains insufficiently investigated.Following AttnGAN, many works reused the same pretrained text encoder, leaving grammar, positional, and numerical information underexamined.
  • 6. Discussion & Challenges: Future research may explore VAEs, autoregressive, flow-based, score-matching, and transformer-based models beyond GANs.The review cautions that IS and FID may be unfair for comparing different generative-model families.

Low Image Resolution of Multiple Object Datasets

The review identifies low-resolution complex-scene datasets, weak scene and object understanding, limited annotations, and inadequate evaluation as barriers to practical T2I systems.

  • Low Image Resolution of Multiple Object Datasets: Complex-scene datasets remain low resolution, and methods still cannot synthesize photorealistic images with multiple interacting objects.The review anticipates a need for high-resolution, diverse multi-object datasets as image quality improves.
  • Low Image Resolution of Multiple Object Datasets: Visually grounded captions, mouse traces, and region-level descriptions could provide finer spatial grounding for textual descriptions.The review also identifies Visual Genome as a possible source of region-associated descriptions.
  • Low Image Resolution of Multiple Object Datasets: Current caption datasets are often better suited to image captioning than to representing subjectivity in human descriptions.Human-generated captions do not commonly describe only obvious image contents, raising questions about dataset requirements for T2I.
  • Low Image Resolution of Multiple Object Datasets: One-sided image-caption annotations permit the same caption to match multiple different images, limiting cross-modal associations.The review notes that multiple matching captions per image could instead support methods such as curriculum learning.
  • Low Image Resolution of Multiple Object Datasets: T2I datasets are largely English-only, motivating multilingual descriptions and models that accept multiple languages without retraining.The review suggests multilingual captions might also benefit generalization.
  • Low Image Resolution of Multiple Object Datasets: Image quality, diversity, and semantic alignment remain difficult to evaluate reliably with existing automatic metrics.Proposed alternatives include detection-based, SceneFID, classification-accuracy, precision-recall, density, and coverage metrics.
  • Low Image Resolution of Multiple Object Datasets: Image-text evaluation lacks a metric set covering multiple aspects, while R-precision can overfit and VS and SOA expose multi-object realism problems.The review states that reliable evaluation still requires user studies.
  • Low Image Resolution of Multiple Object Datasets: Standardized user-evaluation protocols could reduce the variability, expense, and time demands of current studies.The proposed direction includes standardizing image-text alignment studies similarly to HYPE for image quality.

7. Conclusion

The review concludes that T2I has advanced substantially but remains limited for complex scenes and reliable evaluation, with future progress requiring richer supervision, better metrics, and greater control.

  • 7. Conclusion: The review categorizes T2I methods into direct single-caption approaches and methods using additional information such as layouts, masks, scene graphs, or mouse traces.It reports that complex scenes with multiple interacting objects remain very difficult, while semantic-mask methods achieve the best image quality by separating foreground and background generation.
  • 7. Conclusion: Automatic metrics including IS, FID, R-prec., and SOA remain proxies for human judgment and require user-study verification.This is especially important for image-text alignment and subtle numerical or positional information.
  • 7. Conclusion: The review highlights text embeddings, alternative generative models, scene understanding, visually grounded captions, and dense cross-modal associations as open research areas.It also links practical T2I applications to fine-grained, iterative, and interactive manipulation.
  • 7. Conclusion: Further improvement is needed in higher-resolution synthesis, semantic alignment, automatic metrics, standardized user studies, and controllable user interfaces.The review frames these as remaining opportunities despite significant progress.

Appendix A. Collected Results

The appendix compiles reported results across three standard datasets and several evaluation measures, while documenting variation in published scores for the same models.

  • Appendix A. Collected Results: The collected tables cover Oxford-102 Flowers, CUB-200 Birds, and COCO results, plus VS and SOA evaluations.The appendix also includes CIDEr results for COCO.
  • Appendix A. Collected Results: Multiple, often varying scores are reported in the literature for the same model on COCO.Some table rows mark updated results from open-source code, and VS results may use different pretrained evaluation models.
Loading 2101.09983v2…