PaperlayerView exact revision on arXiv

Source-linked AI summary

An Introduction to Image Synthesis with Generative Adversarial Nets

He Huang, Philip S. Yu, Changhu Wang

arXiv:1803.04469v2cs.CV

TL;DR

GAN image synthesis has advanced rapidly, but current methods remain limited in handling complex multi-object scenes and face training instability and mode collapse. This paper organizes the field’s approaches, reviews text-to-image and image-to-image translation models, and identifies practical strengths, weaknesses, and research directions.

  • Problem

    The paper addresses how GAN-based image-synthesis methods should be organized and assessed, given their effectiveness alongside training instability, mode collapse, and limited performance on complex scenes.

  • Method

    The paper provides a taxonomy of direct, hierarchical, and iterative methods and reviews models for text-to-image synthesis and image-to-image translation.

  • Results

    The review finds that supervised Pix2Pix produces sharper images than unsupervised methods, while unsupervised approaches remain promising because they do not require paired data.

  • Takeaways & Limitations

    GANs provide a promising framework for image synthesis, with learned discrimination serving as a learned loss function and image synthesis remaining the most studied application.

  • Takeaways & Limitations

    Current text-to-image models perform poorly when images contain multiple complicated objects, despite reasonable results for single-object images and some scenes.

Abstract

from arXiv · show

There has been a drastic growth of research in Generative Adversarial Nets (GANs) in the past few years. Proposed in 2014, GAN has been applied to various applications such as computer vision and natural language processing, and achieves impressive performance. Among the many applications of GAN, image synthesis is the most well-studied one, and research in this area has already demonstrated the great potential of using GAN in image synthesis. In this paper, we provide a taxonomy of methods used in image synthesis, review different models for text-to-image synthesis and image-to-image translation, and discuss some evaluation metrics as well as possible future research directions in image synthesis with GAN.

1 INTRODUCTION

GANs have become a prominent generative-model framework, with image synthesis the most studied application. This paper focuses on organizing image-synthesis methods and reviewing text-to-image and image-to-image translation.

  • GAN motivation: GANs, proposed in 2014, use a generator to produce realistic samples and a discriminator to distinguish generated from real data.The paper describes GAN research as spanning theoretical and application-oriented threads.
  • Scope: Image synthesis is the paper’s specific focus because it is the most studied area of GAN application.Other computer-vision applications include image in-painting, image captioning, object detection, and semantic segmentation.
  • Broader context: GAN research also extends to NLP tasks such as text modeling, dialogue generation, question answering, and neural machine translation, where training is more difficult.These applications are presented as a growing but challenging research area.
  • Paper organization: The paper classifies image-synthesis methods into direct, hierarchical, and iterative approaches.It also briefly mentions other methods and discusses why GANs perform well in certain tasks.
  • Paper organization: The review gives detailed coverage to text-to-image synthesis and image-to-image translation, alongside evaluation metrics and future research directions.The paper is intended as a guideline for applying GANs and supporting further research.

2 GAN PRELIMINARIES

GANs learn data generation through an adversarial generator–discriminator game, with conditional, encoder-based, and VAE combinations extending the basic framework. The section also covers training instability, mode collapse, and organizational taxonomies for image synthesis.

  • Basic GAN: A GAN generator maps random noise z to synthetic data, while a discriminator estimates whether inputs come from the true or generated distribution.The two networks are trained in a two-player min-max game.
  • Training: The non-saturating generator objective maximizes log D(G(z)) to avoid zero gradients when the discriminator becomes too confident.This changes gradient scale but preserves its direction and avoids saturation.
  • GAN variants: Conditional GANs concatenate an input c with noise z, allowing generated images to depend on class labels, object attributes, or other information.The conditional input can be augmented together with the noise vector.
  • GAN variants: Auxiliary classifiers add task-specific supervision to the discriminator, supporting semi-supervised learning and potentially sharper images with less mode collapse.The classifier predicts labels for generated images from the conditional architecture.
  • GAN variants: BiGAN and ALI add an encoder E that maps data samples into latent features, enabling an inverse mapping unavailable in the original GAN.Their discriminator receives both a data sample and a feature vector.
  • GAN variants: VAE-GAN combines VAE regularization and reconstruction with GAN training to seek both image sharpness and variety.Experiments reported that VAE-GAN generated better images than VAE or GAN alone.
  • Failure modes: GAN training is unstable and can suffer mode collapse, where the generator produces limited varieties of samples despite realistic individual outputs.The paper distinguishes inter-class collapse from intra-class collapse using digit identities and writing styles as examples.
  • Method taxonomy: Image-synthesis methods are grouped as direct, hierarchical, or iterative according to their use and organization of generators, discriminators, and resolutions.Direct methods use one generator and discriminator, whereas hierarchical and iterative methods use multiple GANs.

3 GENERAL APPROACHES OF IMAGE SYNTHESIS WITH GAN

The paper groups GAN image-synthesis methods into direct, hierarchical, and iterative approaches. These methods differ in generator organization, refinement strategy, and use of weight sharing.

  • Direct Methods: Direct methods use one generator and one discriminator with straightforward, branch-free structures.They are relatively straightforward to design and implement and usually achieve good results.
  • Hierarchical Methods: Hierarchical methods use two generators and two discriminators to separate image components such as style and structure or foreground and background.The generators may operate in parallel or sequentially.
  • Iterative Methods: Iterative methods use multiple similar generators to refine images from coarse to fine, often sharing weights when generator structures match.Each generator refines details from the preceding generator.
  • Iterative Methods: LAPGAN generates residual details through a Laplacian pyramid and outperforms the original GAN, producing sharper images than the direct method.Its lowest-level generator takes noise, while later generators add residual images to preceding outputs.
  • Iterative Methods: SGAN stacks generators that transform lower-level features into higher-level features, with separate auxiliary networks constraining each feature level.Each generator is associated with an encoder, discriminator, and Q-network.
  • Iterative Methods: GRAN applies the same generator recurrently, feeding each output into the next step and sharing weights across all steps.This weight-sharing design resembles a classic recurrent neural network.

4 TEXT-TO-IMAGE SYNTHESIS

Text-to-image synthesis conditions GAN generation on textual descriptions, but current models remain limited in spatial control and complex multi-object scenes. The reviewed methods use text embeddings, stacked generation, location constraints, attention, or iterative sampling.

  • Overview: Text-to-image synthesis seeks to generate images from text descriptions, a challenging goal beyond class-conditioned generation.The paper characterizes realistic text-to-image generation as a major computer-vision challenge.
  • Text-Conditioned Generation: GAN-INT-CLS uses sentence embeddings with noise as conditioning and trains the discriminator on matched, mismatched, and generated image-text pairs.This distinguishes unrealistic images from mismatched text-image combinations.
  • Text-Conditioned Generation: TAC-GAN combines GAN-INT-CLS and AC-GAN and achieves a higher Inception Score than GAN-INT-CLS and StackGAN on Oxford-102.The comparison is reported specifically for the Oxford-102 dataset.
  • Location Constraints: GAWWN adds location constraints through either learned bounding boxes or user-specified keypoints processed as spatial masks.Its bounding-box approach is end-to-end and does not require additional input.
  • Stacked GANs: StackGAN, StackGAN++, and AttnGAN produce sharper images than GAN-INT-CLS and GAWWN on CUB and Oxford-102.AttnGAN also reports a higher Inception Score than PPGN on COCO, although its examples do not appear visually better.
  • Iterative Sampling: PPGN uses iterative activation-maximization sampling with a pretrained captioning model and a separately trained image generator.The image generator combines a denoising autoencoder and GAN.
  • Iterative Sampling: PPGN takes longer to generate images but achieves higher-resolution, better-quality results than earlier methods and performs among the best in class- and text-conditioned synthesis.The paper explicitly identifies the test-time cost as a trade-off.
  • Limitations: Current text-to-image models perform well mainly on single-object datasets but work badly when images contain multiple complicated objects.Scenes such as bedrooms and living rooms can be reasonable but lack sharp details.

5 IMAGE-TO-IMAGE TRANSLATION

Image-to-image translation maps one representation of a scene to another, including BW-to-RGB conversion. The reviewed methods combine adversarial objectives with reconstruction or pairwise-discrimination mechanisms, with paired supervision remaining a key constraint.

  • Overview: Image-to-image translation converts one scene representation into another, such as BW images into RGB images.The paper relates this task to style transfer and describes it as a generalization of that problem.
  • Pix2Pix: Pix2Pix combines conditional GAN loss with L1 regularization so generated images both fool the discriminator and remain close to ground truth.The paper states that L1 produces less blurry images than L2.
  • Pix2Pix: Pix2Pix’s objective balances conditional GAN and L1 losses through the hyper-parameter λ.The overall objective combines the two loss terms.
  • Pix2Pix: Pix2Pix replaces explicit random noise with dropout because noise had no obvious effect on its output.Dropout is used during both training and testing.
  • Pix2Pix: Its U-Net generator uses skip connections to preserve low-level information such as object edges across the encoder-decoder bottleneck.The discriminator uses PatchGAN, which averages scores over N × N image patches.
  • Pix2Pix: Pix2Pix requires paired images as supervision, with source and target images drawn from a joint distribution.This paired-data requirement is identified as its major limitation.
  • PLDT: PLDT adds a pair discriminator that judges whether images from different domains are associated, alongside the generator and other discriminator.Its architecture is illustrated for source, ground-truth target, irrelevant target, and generated target images.
  • PLDT: PLDT modifies object geometry while maintaining consistent texture among associated images.Its generator uses an encoder-decoder design, while both discriminators are fully convolutional.

5.3 Unsupervised Image-to-Image Translation with Cyclic Loss

CycleGAN and DualGAN perform unsupervised image-to-image translation with two opposing generators, two domain discriminators, and cycle-consistency losses that reconstruct inputs after round-trip translation. CycleGAN can produce high-quality translations from unpaired data, but supervised Pix2Pix performs better and geometric transformations remain difficult.

  • Cycle-consistency minimizes reconstruction error after translating an image to the other domain and back in either direction.For example, a is mapped through GAB and GBA to approximate a again; the objective combines adversarial and cycle-consistency terms.
  • CycleGAN and DualGAN use opposing generators and one discriminator per domain to translate between domains without paired examples.The generators implement A→B and B→A transformations, while the discriminators judge domain membership.
  • CycleGAN uses least-square adversarial loss and discriminator updates based on 50 previously generated images to stabilize training.These techniques replace the log loss and reduce model oscillation, respectively.
  • CycleGAN produces high-quality image-to-image translations using unpaired data, although supervised Pix2Pix still outperforms it by a noticeable margin.Experiments also show that both cycles matter for the cycle-consistency loss.
  • CycleGAN fails on transformations requiring geometric changes, including apple↔orange and cat↔dog.The reported failure cases indicate a boundary for the approach’s demonstrated capability.

5.4 Unsupervised Image-to-Image Translation with Distance Constraint

Distance-constrained translation methods preserve relationships between images by encouraging source- and target-domain pairwise distances to correspond. DistanceGAN improves over cyclic baselines in several tasks, but the review notes unresolved choices about distance objectives and combinations.

  • DistanceGAN observes a strong positive correlation between pairwise source-image distances and distances between their translated counterparts.This motivates preserving relative differences across domains during translation.
  • DistanceGAN minimizes normalized absolute differences between source and translated pairwise distances rather than maximizing their product.Maximizing the product lets large source-domain distances dominate the loss.
  • DistanceGAN adds a self-distance constraint that supports stochastic gradient descent when only one sample is fed into the model at a time.The constraint computes statistics from corresponding left or right image halves.
  • One-sided distance or self-distance losses outperform DiscoGAN and CycleGAN in several tasks, while combining cyclic and distance losses achieves the best results in some cases.The experiments compare cyclic, one-sided distance, combined, and one-sided self-distance objectives.
  • DistanceGAN computes distances in raw RGB space, although latent feature-space distances from pretrained classifiers may be beneficial.The review presents this as a possible improvement rather than an established result.
  • The review proposes an ordering-based triplet loss because the concept of a translated distance being “high” does not specify a definite target value.The alternative preserves relative ordering rather than requiring an absolute distance magnitude.

5.5 Unsupervised Image-to-Image Translation with Feature Constancy

Feature-constancy methods constrain unsupervised translation by preserving higher-level features, while also encouraging target-domain identity mappings and using adversarial discrimination. DTN produces impressive face-to-emoji images competitive with existing emoji-generating programs.

  • DTN decomposes its generator into a convolutional feature extractor and a transposed-convolutional network, G = g ◦ f.The feature extractor supports reconstruction constraints at a higher feature level.
  • DTN preserves high-level features of source images after translation by minimizing feature reconstruction error.For an input x from the source domain, the translated output is compared through a distance measure; DTN uses MSE.
  • DTN encourages the generator to act as an identity mapping for images already in the target domain.This target-domain identity constraint uses MSE as its distance measure.
  • DTN replaces the regular binary discriminator with a ternary classifier distinguishing transformed source images, transformed target images, and untransformed target images.The three classes are explicitly assigned to these source and target cases.
  • DTN’s generator objective combines adversarial, feature-constancy, target-identity, and total-variation losses.The total-variation term is added to slightly smooth generated images.
  • DTN produces impressive face-to-emoji images competitive with some existing emoji-generating programs.

5.6 Unsupervised Image-to-Image Translation with Auxiliary Classifier

Auxiliary-classifier approaches add task-specific labels or content constraints to unsupervised image-to-image translation, while UNIT combines VAE and GAN objectives under a shared-latent-space assumption. Their reported benefits are accompanied by limits in label availability and evaluation coverage.

  • DAAC uses a task-specific classifier to assign labels to images from either source or target domains during unsupervised translation.The classifier can be an image-classification model.
  • DAAC adds content-similarity losses when prior knowledge specifies information that should remain unchanged after domain adaptation.One example preserves object content while changing black backgrounds.
  • Auxiliary classifiers can improve adapted-image quality and help avoid mode collapse, but real-world labeled data and fine-grained pixel masks are limited.
  • UNIT adds VAE components to CoGAN and assumes that corresponding images from two domains share a latent space.It enforces this assumption through weight sharing in encoder and generator layers.
  • UNIT outperforms DTN and CoGAN on MNIST and SVHN cross-domain classification accuracy, but it is not compared with CycleGAN or DiscoGAN.The review also notes that widely used metrics such as Inception Score are omitted.

5.8 Unsupervised Multi-domain Image-to-Image Translation

The section reviews supervised and unsupervised image-to-image translation losses, then surveys applications including face editing, video prediction, and super-resolution. It highlights Pix2Pix’s sharp images, the promise of unpaired methods, and video-quality limitations over longer horizons.

  • Unsupervised Multi-domain Translation: StarGAN uses one generator conditioned on a target-domain label to translate images among several domains, avoiding a separate generator for each domain pair.The multi-domain design addresses the cost of training separate generators for every domain pair.
  • Translation Losses: Pixel-wise L1 reconstruction requires paired samples, whereas one-sided and bidirectional reconstruction losses enforce cycle-consistency without paired data.An additional VAE loss assumes a shared latent space between source and target domains.
  • Translation Losses: Discriminator-side losses improve discrimination between real and fake samples, allowing the generator to learn from an enhanced discriminator.The reviewed losses include pair-wise discriminator, ternary discriminator, and auxiliary classifier losses.
  • General Findings: Pix2Pix produces the sharpest images among the reviewed models, while unsupervised methods are less sharp but avoid costly paired-data collection.The paper suggests combining L1 loss with pair-wise discrimination and preserving source-target similarity as possible improvements.
  • Image Super-Resolution: SRGAN combines adversarial, pixel-wise MSE, perceptual, and regularization losses for single-image super-resolution, outperforming some baselines by small metric margins.Differences in synthetic image quality are difficult to see without zooming in.
  • Video Prediction: Both reviewed video-prediction methods produce reasonable short-interval videos, but quality worsens with longer intervals as objects lose their shapes.The paper attributes this possible failure to difficulty learning legal object movements.

6 EVALUATION METRICS ON SYNTHETIC IMAGES

Synthetic-image quality is difficult to quantify because real and generated images lack one-to-one correspondence and human judgments vary. The section compares classifier-based metrics with FID, emphasizing limitations of IS and the stronger reported properties of FID.

  • Motivation: Synthetic-image quality is difficult to quantify because RMSE assumes one-to-one correspondence, while human ratings can vary across evaluators.AMT is a commonly used subjective evaluation method.
  • Inception Score: Inception score rewards confident class predictions and diversity across generated classes, but it is insensitive to label priors and vulnerable to intra-class mode collapse.A model can obtain a perfect score by generating one perfect sample for each class.
  • Classifier-based Metrics: Classifier-based metrics may overlook small visual-detail differences because correct classification does not require sharp images.Classifier decisions can also be influenced by noise invisible to humans, as shown by adversarial-example research.
  • Fréchet Inception Distance: FID embeds real and generated images, models their features as multivariate Gaussians, and compares the resulting means and covariances with Fréchet distance.The compared statistics are estimated from embeddings in a chosen Inception-Net layer.
  • Metric Comparison: FID is reported as consistent with human judgment, negatively correlated with generated-image quality, less noise-sensitive than IS, and able to detect intra-class mode collapse.Although IS is more widely adopted, the paper reports that FID has been shown to be better than IS.

7 DISCRIMINATORS AS LEARNED LOSS FUNC-

GANs use the discriminator as a learned loss function rather than relying only on task-specific, explicitly designed losses. This is especially relevant when image-synthesis objectives are difficult to express mathematically.

  • Implications: Learned evaluation can make image-synthesis models more flexible and potentially improve generalization when explicit task losses are difficult to design.Style transfer is given as an example where matching a painting style is hard to encode as a mathematical loss.
  • Learned Losses: A discriminator learns to evaluate how well generated outputs match the true data distribution, complementing losses based on ground-truth correspondence.The paper contrasts L(x, x̂), which measures closeness to a corresponding target, with L(x), learned from the data distribution.

8 DISCUSSION AND CONCLUSION

The paper organizes GAN-based image synthesis into three main approaches and reviews text-to-image and image-to-image translation methods, applications, and evaluation metrics. It finds strong results on simple text-to-image datasets but substantially weaker performance on complex scenes, while emphasizing learned discriminator losses.

  • Contributions: The paper classifies image-synthesis methods as direct, hierarchical, or iterative and discusses text-to-image synthesis and image-to-image translation.It also mentions other generation methods such as iterative sampling.
  • Text-to-Image Synthesis: Text-to-image methods work well for single-object datasets but perform much worse on complex datasets such as MSCOCO.The paper links this limitation to difficulty learning different object concepts and proposes separate object-generation and composition models as one possible direction.
  • Image-to-Image Translation: Image-to-image translation is reviewed from supervised to unsupervised settings, alongside task-specific models for face editing, video prediction, and image super-resolution.The section describes pixel-wise, cyclic, and self-distance losses among the general methods.
  • Evaluation and Perspective: The paper reviews synthetic-image metrics and argues that GANs’ discriminator can act as a learned loss for outputs whose quality is hard to evaluate with explicit equations.This learned evaluation is presented as a central source of GAN power for image-synthesis tasks.
Loading 1803.04469v2…