PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Synthesis with Adversarial Networks: a Comprehensive Survey and Case Studies

Pourya Shamsolmoali, Masoumeh Zareapoor, Eric Granger, Huiyu Zhou, Ruili Wang, M. Emre Celebi, Jie Yang

arXiv:2012.13736v1cs.CVeess.IV

TL;DR

Existing GAN research lacked a comprehensive review covering loss variants, evaluation metrics, diversity, and training stability. This survey organizes GAN methods for image synthesis by architectures, constraints, losses, metrics, datasets, and application categories, combines comparative and theoretical discussion, and provides a living repository of related code, datasets, and papers. It concludes that GAN image synthesis remains difficult in areas including explicit loss design, domain adaptation, transfer learning, and broader evaluation.

  • Problem

    Prior GAN surveys did not comprehensively analyze model advantages, drawbacks, theoretical foundations, loss variants, evaluation metrics, diverse generation, and stable training.

  • Method

    The survey organizes GAN image-synthesis research by application, architecture, learning strategy, loss function, evaluation metric, dataset, and training considerations, alongside empirical comparative analysis.

  • Results

    The paper presents a comprehensive review of GAN variants, architectures, losses, datasets, metrics, and image-synthesis applications, supported by an empirical comparative study and a living repository.

  • Takeaways & Limitations

    The survey provides a structured reference for comparing GAN approaches and locating implementations, datasets, and papers for synthetic image generation.

  • Takeaways & Limitations

    The study lacks a thorough analysis of domain adaptation and transfer learning and calls for comparisons that include training and testing time alongside accuracy.

Abstract

from arXiv · show

Generative Adversarial Networks (GANs) have been extremely successful in various application domains such as computer vision, medicine, and natural language processing. Moreover, transforming an object or person to a desired shape become a well-studied research in the GANs. GANs are powerful models for learning complex distributions to synthesize semantically meaningful samples. However, there is a lack of comprehensive review in this field, especially lack of a collection of GANs loss-variant, evaluation metrics, remedies for diverse image generation, and stable training. Given the current fast GANs development, in this survey, we provide a comprehensive review of adversarial models for image synthesis. We summarize the synthetic image generation methods, and discuss the categories including image-to-image translation, fusion image generation, label-to-image mapping, and text-to-image translation. We organize the literature based on their base models, developed ideas related to architectures, constraints, loss functions, evaluation metrics, and training datasets. We present milestones of adversarial models, review an extensive selection of previous works in various categories, and present insights on the development route from the model-based to data-driven methods. Further, we highlight a range of potential future research directions. One of the unique features of this review is that all software implementations of these GAN methods and datasets have been collected and made available in one place at https://github.com/pshams55/GAN-Case-Study.

1. Introduction

This survey addresses the need for a comprehensive, theoretically informed review of GANs for synthetic image generation. It organizes models, applications, metrics, datasets, and training considerations while providing comparative analysis and a living repository.

  • Research gap: The survey identifies a gap in prior reviews: they largely cover GAN architectures and algorithms without comprehensively analyzing each model’s advantages, drawbacks, and theoretical foundations.It frames this gap as motivating a more detailed survey of GANs for image synthesis.
  • Research questions: It examines open questions about state-of-the-art GANs, efficient synthetic-image generation, architectures for image-to-image translation, effective loss functions, and evaluation datasets.These questions define the survey’s scope and analytical priorities.
  • Comparative study: The paper reports an empirical comparative study using a common Python and TensorFlow framework to train GAN models for synthetic image generation.The comparison used four NVIDIA GTX Geforce 1080 Ti GPUs.
  • Organization: The review organizes recent GAN applications for synthetic image generation into single-stage and multi-stage models and reviews architectures of state-of-the-art systems.The taxonomy and architecture review provide a structured view of the literature.
  • Evaluation: It provides details on performance metrics and datasets commonly used to evaluate GAN models.These evaluation resources are presented as part of the survey’s coverage of GAN research.
  • Resources: A continuously updated repository collects the source codes, datasets, and papers discussed in the survey.The repository is intended as a live resource for the community.

2. Related Work

The reviewed GAN literature spans architectures, training strategies, applications, and evaluation, while highlighting persistent challenges such as mode collapse, vanishing gradients, and convergence. This section also contrasts conditional, auto-encoder, and auxiliary-classifier variants for image synthesis.

  • GAN research includes performance validation, but benchmark datasets may inadequately reflect diversity, causing evaluations to focus primarily on image quality assessment.
  • GAN training faces mode collapse, vanishing gradients, convergence difficulties, and imbalance between generator and discriminator networks.
  • Conditional GANs support class-conditional synthesis, while InfoGAN and ACGAN use related architectures to incorporate semantic or auxiliary-classification information.
  • Auto-encoder GANs use discriminator reconstruction loss, whereas BiGAN adds an encoder to map generated data back to latent space.
  • Auxiliary-classifier GANs are reported to generate sharper edge images, handle collapse, and perform in image-to-image translation and text-to-image synthesis.

3. GANs and Loss-Variants

GAN loss variants address unstable training and distribution-matching problems, with reported differences in mode coverage between regular GANs and LSGANs. The section also reviews theoretical divergences and methods targeting missing modes, gradients, and convergence.

  • Changing GAN loss functions has improved training optimality and convergence, although unstable training remains a major issue.
  • The optimized discriminator defines the generator objective through probability distributions of real and generated data and divergences including KL and JS.
  • Regular GANs suffer from mode collapse in Gaussian-kernel experiments, whereas LSGAN learns the Gaussian mixture distribution successfully.
  • Loss-sensitive GAN separates real and generated samples by a divergence margin to improve realism and address model collapse and vanishing gradients.
  • AdaGAN adds components through GAN training on reweighted samples, and experiments report that this progressive procedure addresses missing modes.

4. Datasets

The survey catalogs datasets used to evaluate GAN approaches, spanning handwritten digits, natural images, faces, objects, street scenes, videos, and artwork. These datasets differ in size, modality, annotation structure, and class coverage.

  • The reviewed evaluation datasets span object recognition, faces, scenes, videos, artwork, digits, clothing, shoes, and synthetic or stylized images.
  • MNIST contains 60,000 training and 10,000 testing handwritten-digit images, each represented as a 28×28 image or a 784-dimensional vector.
  • Fashion MNIST contains 60,000 training and 10,000 testing grayscale images across 10 classes.
  • CIFAR-10 provides 50,000 training and 10,000 testing 32×32 color images across 10 classes, while CIFAR-100 has 100 classes with 500 training and 100 testing images per class.
  • CelebA contains more than 200,000 celebrity face images with 40 attribute annotations and substantial pose and background variation.
  • Cityscapes combines 5,000 densely annotated video frames with 20,000 weakly annotated frames from street scenes.
  • Other listed resources include Toronto Faces, ImageNet, Van Gogh paintings, DSLR pairs, Caltech-UCSD Birds, SVHN, FaceScrub, Paris StreetView, YouTubeFace, CartoonSet, and SENSIAC.
  • UT Zappos50K contains 50,025 shoe catalogs divided into broad footwear categories, functional types, and brands.

5. Synthetic Image Generation Methods

Synthetic image generation methods use learned models to create new samples from existing training data, supporting photo-realistic reproduction and automated data augmentation.

  • Synthetic data methods generate new sample images from actual training sets, including photo-realistic reproduction, learning-based generation, and automated data augmentation.

5.1. Single-stage methods:

Single-stage GAN methods use one generator and one discriminator with simple networks, while surveyed variants add architectural or representational mechanisms to improve synthesis and control.

  • Single-stage approaches use one generator and one discriminator without additional connections.
  • DCGAN introduces architectural constraints and deep convolutional networks that provide efficient and accurate unsupervised learning.
  • ControlGAN separates the discriminator from the classifier to manage the random distribution and detailed content of generated samples.
  • Decoder-encoder GANs combine adversarial training with variational Bayesian inference to improve generated-image quality.

5.2. Multi-stage methods:

Multi-stage GAN methods use multiple generators and discriminators to divide synthesis into distinct tasks, often separating structure, style, foreground, or background. The reviewed applications span face style transfer, text-to-image generation, video synthesis, and haze removal, while raising concerns about efficiency, evaluation, and output quality.

  • Multi-stage architectures: Multi-stage architectures assign different tasks to multiple generators and discriminators, operating either in parallel or sequentially.They may separate foreground from background or styles from structure.
  • Face style transfer: Face style transfer uses an attribute image and an additional face-verification network to produce a conditional vector for high-quality multi-path generation.The method is illustrated with CelebA samples.
  • Text-to-image synthesis: StackGAN uses two generator stages: the first produces a coarse, blurry image from noise and text, while the second adds more realistic details.The survey notes improved accuracy but questions the quantification of speed improvements.
  • Other applications: Other reviewed multi-stage systems generate high-resolution videos, preserve object and background information, or combine pixel-wise, adversarial, and perceptual losses for haze removal.The haze-removal results can retain fine textures, but deformation and noise remain.
  • Evaluation and limitations: The surveyed methods include computationally demanding systems and studies with limited experiments or incomplete comparisons against state-of-the-art models.These concerns limit conclusions about efficiency and relative performance.

5.3. Applications to medical imaging:

GANs support medical imaging through generative data synthesis and discriminative anomaly or prior learning, addressing data scarcity, privacy, segmentation, and image translation. Applications include CT and MRI augmentation, anomaly detection, and cross-contrast synthesis, but the supplied passages provide no unified performance result.

  • Medical imaging roles: Medical imaging GANs are used either to generate realistic training data or to make the discriminator function as a learned prior.The generative use can help address data scarcity and patient privacy.
  • Evidence scope: The supplied medical-imaging passages describe applications and comparisons but do not establish a single overall performance conclusion.The cited material reports methods rather than a common benchmark outcome.
  • Segmentation and anomaly detection: CycleGAN-based augmentation is used to improve generalizability in CT segmentation, while two-stage GANs support unsupervised anomaly detection in MRI scans.These methods target different medical-imaging tasks rather than one common pipeline.
  • MR image translation: Unsupervised CycleGAN and UNIT models are compared for translating T1- and T2-weighted MR images against real images.The comparison evaluates the created synthetic MR images relative to real images.
  • CT reconstruction and segmentation: A stacked GAN for CT segmentation first reduces image noise and then generates a higher-resolution image with enhanced boundaries.The two stages separately target denoising and boundary enhancement.

5.4. Applications to 3D Reconstruction

GAN-based 3D reconstruction methods recover missing or complete object geometry from low-resolution, depth-view, or image inputs. The reviewed approaches combine encoder-decoder, volumetric, recurrent, probabilistic, and iterative designs, with evaluation limitations noted for some models.

  • 3D completion: 3D-ED-GAN combines a 3D encoder-decoder GAN with a recurrent convolutional network to fill missing data in low-resolution 3D inputs.The associated figure presents low-resolution completion after processing through LRCN.
  • Probabilistic reconstruction: 3D-VAE-GAN maps a low-dimensional probabilistic space to 3D objects and reconstructs a 3D object directly from a 2D input image.Its structure is described as simple, but its evaluation results and parameters are limited.
  • Single-view reconstruction: 3D-RecGAN reconstructs a complete 3D occupancy grid from one depth-view voxel representation by filling missing regions.Unlike methods requiring multiple views or class labels, it uses a single random depth view.
  • Iterative and loss-based designs: Iterative GANs transform an input image into an output image using object geometry and appearance, while another 3D model combines completion loss with an Improved Wasserstein GAN loss.The iterative model was evaluated on several datasets but did not generate highly realistic samples.

5.5. Image fusion

GAN-based image fusion combines input images into a new image while changing shape or characteristics and preserving their main content. FusionGAN, SF-GAN, and related methods target realistic synthesis from multiple input sources.

  • FusionGAN: FusionGAN generates a fusion image by manipulating two input images while preserving their main content and changing input shape or characteristics.The method is presented as a GAN-based framework for generating a new image from input images.
  • SF-GAN and related methods: SF-GAN synthesizes realistic images from foreground objects and background images, with effectiveness demonstrated through a comprehensive set of experiments.The survey also lists additional GAN architectures for transforming inputs into desired shapes and improving fusion.

5.6. Image Completion

Image completion fills missing or masked image regions with synthesized content that fits the surrounding context and can resemble the original. A double-path framework combines reconstruction-based priors with a generative path, but such methods often struggle with human-body structure and appearance.

  • Image Completion: Image completion synthesizes missing or masked regions with detailed content that fits the image context.Many conventional methods use neighboring patches and low-level cues to create plausible content.
  • Image Completion: A double-path framework combines a reconstructive path for estimating missing-part priors with a generative path conditioned on that distribution.The reconstructive path uses a single ground-truth image to recreate the input before informing generation.
  • Image Completion: Human-body images remain difficult because accurate structure and appearance synthesis can exceed these completion methods’ capabilities.The cited work reports that such methods often fail in this setting.

6. Image-to-Image-Translation

Image-to-image translation maps images between domains, with supervised methods using paired examples and unsupervised methods learning from separate unpaired sets. The reviewed approaches combine adversarial, reconstruction, cycle-consistency, distance, and stabilization objectives across diverse translation tasks.

  • Image-to-Image-Translation: Image-to-image translation maps an image from one domain to another, including tasks such as super-resolution and colorization.Supervised approaches use paired images, whereas unsupervised approaches use separate image sets without corresponding examples.
  • Image-to-Image-Translation: Unsupervised translation is more difficult because inferring a joint cross-domain distribution from separate marginal distributions is ill-posed.The review also notes that unpaired data collection is more feasible than collecting corresponding images.
  • Supervised Translation: Conditional adversarial translation combines GAN loss with L1 regularization so generated images both pass discriminator filtering and resemble ground-truth images.The reviewed method uses L1 because it produces less blur than L2 and balances the losses with λ.
  • Supervised Translation: Supervised translation can add a secondary discriminator that evaluates whether source and target images form a related pair.The generator transfers a source-domain image into an associated target-domain image.
  • Unsupervised Translation: Unsupervised cyclic models use reverse generators and discriminators to encourage translated images to reconstruct the original domain.The cycle maps an image through both domains and minimizes reconstruction error; λ balances the losses.
  • Applications and Comparisons: The reviewed translation literature spans street scenes, gender transformation, cat-to-species translation, face aging, and tabulated loss-function and architecture comparisons.The cited figures show input-to-output examples for several tasks, while Table 3 organizes methods by input, output, characteristics, losses, and backbone code.
  • Unsupervised Translation: Training stabilization includes least-square adversarial loss and discriminator updates using previously generated images.The review associates these choices with more stable training and more realistic generated images, while also discussing face aging and distance-preserving objectives.

7. Conclusion and Discussion

The survey reviews GAN variants for synthetic image generation by architecture, performance, and training stability, while discussing methods’ strengths and weaknesses. It identifies unresolved needs in domain adaptation, transfer learning, efficiency comparisons, normalization effects, and applications beyond image synthesis.

  • Conclusion and Discussion: The survey reviews GAN architectures, performance, stable training, loss functions, datasets, and recent supervised and unsupervised synthesis methods.It also discusses the strengths and weaknesses of the reviewed approaches.
  • Conclusion and Discussion: Explicitly defining losses remains difficult because synthetic image tasks can admit several legitimate outputs, such as different valid styles.The review highlights style transfer as an example where matching an image to a desired style is difficult to evaluate with a loss.
  • Future Directions: The survey identifies insufficient study of domain adaptation and transfer learning for GAN-based synthetic image generation.It also calls for comparisons including training and testing time, further study of normalization effects, and research on video, time series, and natural language generation.
Loading 2012.13736v1…