PaperlayerView exact revision on arXiv

Source-linked AI summary

A review of Generative Adversarial Networks (GANs) and its applications in a wide variety of disciplines -- From Medical to Remote Sensing

Ankan Dash, Junyi Ye, Guiling Wang

arXiv:2110.01442v1cs.LGcs.AIcs.CV

TL;DR

The paper surveys GAN models and their applications across diverse domains, including challenging settings such as financial data modeling. It presents GAN theory, applications, and open research topics, while identifying mode collapse as a limitation.

  • Problem

    Financial data modeling is challenging because financial data have complex statistical properties and dynamic stochastic behavior.

  • Method

    The paper presents state-of-the-art GAN models, their theory, applications across diverse domains, and open research topics.

  • Results

    The paper provides a survey of GAN applications across a wide variety of domains and discusses theory, applications, and open research topics.

  • Takeaways & Limitations

    The study is intended to help researchers understand GANs and apply them in their own fields.

  • Takeaways & Limitations

    Mode collapse can cause the Generator to produce only a single type of output instead of a wide range of outputs.

Abstract

from arXiv · show

We look into Generative Adversarial Network (GAN), its prevalent variants and applications in a number of sectors. GANs combine two neural networks that compete against one another using zero-sum game theory, allowing them to create much crisper and discrete outputs. GANs can be used to perform image processing, video generation and prediction, among other computer vision applications. GANs can also be utilised for a variety of science-related activities, including protein engineering, astronomical data processing, remote sensing image dehazing, and crystal structure synthesis. Other notable fields where GANs have made gains include finance, marketing, fashion design, sports, and music. Therefore in this article we provide a comprehensive overview of the applications of GANs in a wide variety of disciplines. We first cover the theory supporting GAN, GAN variants, and the metrics to evaluate GANs. Then we present how GAN and its variants can be applied in twelve domains, ranging from STEM fields, such as astronomy and biology, to business fields, such as marketing and finance, and to arts, such as music. As a result, researchers from other fields may grasp how GANs work and apply them to their own study. To the best of our knowledge, this article provides the most comprehensive survey of GAN's applications in different fields.

1 INTRODUCTION

GANs are generative models in which a Generator and Discriminator compete to produce synthetic data resembling real data. This review expands beyond computer vision to survey GAN variants, evaluation metrics, applications across 12 domains, challenges, and future directions.

  • GAN foundations: GANs learn a probability distribution from training data and generate new samples resembling the real data distribution.The Generator produces samples while the Discriminator distinguishes actual from generated samples.
  • GAN applications: Their applications include data generation, augmentation, image-to-image translation, and image super resolution.
  • Review scope: The paper focuses on GAN applications across engineering, science, social science, business, art, music, and sports, which prior reviews covered less broadly.It presents this as the first review to cover applications across such diverse domains.
  • Review scope: The review addresses GAN theory, widely used variants, evaluation metrics, applications, training challenges, and future research directions.Applications span image and video tasks, healthcare, biology, astronomy, remote sensing, material science, finance, marketing, fashion, sports, and music.
  • Impact: The review aims to help researchers from varied backgrounds understand GAN operation and discover their broad applications.
  • Evaluation: The survey includes quantitative evaluation metrics for both computer vision and time-series data analysis.The authors state that time-series evaluation metrics were not discussed in other GAN survey papers.

2 GAN, GAN VARIANTS AND EXTENSIONS

This section introduces GANs as adversarial generative models and surveys variants that add conditioning, alternative distances, convolutional architectures, progressive training, and specialized generation mechanisms.

  • GAN foundations: GANs use a Generator and Discriminator in a minimax game: the Generator produces samples while the Discriminator distinguishes real from generated data.Both networks are updated during adversarial training.
  • GAN foundations: The standard GAN pipeline feeds random noise to the Generator, evaluates generated and real samples with the Discriminator, and feeds its probabilities back to training.The Generator parameters are updated using Discriminator feedback.
  • Conditional GANs: cGANs condition both networks on additional information such as class labels, giving control over the modes of generated data.The extra information is concatenated with the input to both the Generator and Discriminator.
  • Wasserstein GANs: WGAN replaces the traditional distribution distance with Wasserstein distance, while WGAN-GP uses a gradient penalty instead of weight clipping.The Wasserstein objective seeks generated distributions close to the real distribution, and the reported algorithm improved stability and prevented mode collapse.
  • Convolutional and progressive variants: DCGAN uses convolutional networks and architectural constraints to achieve stable training and generate complex, high-quality outputs.Its design includes transposed convolutions in the Generator, strided convolutions, batch normalization, specified activations, and Adam optimization.
  • Specialized GAN variants: Progressive GANs synthesize high-resolution images by increasing resolution gradually, while CycleGAN enables unpaired translation through cycle consistency.StyleGAN targets diverse, high-resolution facial images; RGAN and RCGAN generate realistic time-series data while capturing temporal dynamics.

3 GAN EVALUATION METRICS

GAN evaluation remains difficult because generated samples must be assessed for both quality and diversity. The survey presents metrics for images and synthetic sequences, while noting gaps in quantitative assessment.

  • Evaluation challenges: GAN evaluation is challenging because generated images must be judged for meaningful content and diversity.The survey frames performance assessment as a persistent difficulty.
  • Image metrics: Higher IS indicates generated samples that are both high quality and diverse.IS favors low conditional-label entropy and high marginal-label entropy.
  • Image metrics: Mode Score addresses IS’s disregard for ground-truth label priors by comparing real and generated distributions.The survey reports that it measures variety and related distributional properties.
  • Image metrics: FID quantifies differences between real and generated image distributions in Inception feature space, with lower values indicating closer distributions.It compares Gaussian summaries of feature embeddings using Fréchet distance.
  • Sequence metrics: Sequence evaluation examines diversity, fidelity, and usability using visualization, discriminative scores, MMD, and time-dependence measures.A discriminative classifier struggles when synthetic and original data appear to share a distribution; MMD² is reported as more informative than generator or discriminator loss and correlates with visual quality.

4 GAN APPLICATIONS

The survey covers GAN applications across image processing and video generation. These applications include image enhancement and manipulation, unconditional and conditional video synthesis, and video prediction.

  • Application scope: GANs are applied across image processing, video generation and prediction, and numerous scientific, business, and creative domains.The survey lists image processing, video, medical and healthcare, biology, astronomy, remote sensing, material science, finance, fashion, sports, and music.
  • Image processing: Image-processing applications include super-resolution, de-raining, inpainting, face generation, and facial-attribute manipulation.Examples include SRGAN for 4x up-scaling, StyleGAN for high-quality diverse faces, and models that alter expressions, hairstyles, or eyeglasses.
  • Video generation: Conditional and unconditional video generation differ in whether outputs use input signals such as text, audio, or speech.The survey separates video synthesis into unconditional generation, conditional generation, and video prediction.
  • Video generation: Video GANs use specialized components to model spatial appearance, temporal dynamics, motion, and texture.TGAN uses temporal and image generators, while FTGAN separates motion through FlowGAN from texture through TextureGAN.
  • Video generation: Early GAN-based video-generation outputs were typically low quality during training, motivating architectures such as DVD-GAN for spatial and temporal modeling.DVD-GAN combines recurrent units with dual discriminators for high-quality frame generation.
  • Video generation: Text-to-video systems condition a gist and subsequent video generation on encoded text to represent layout, content, and motion.The conditional gist generator provides a general representation before cGAN-based video generation.

4.3 Medical and Healthcare

GANs support medical image translation, synthesis, augmentation, and super-resolution where imaging data or acquisition quality can be limited. Reported applications include improved recognition, classification, and image enhancement, alongside trade-offs between numerical and visual quality.

  • Applications and motivation: Medical GAN applications include image translation, synthetic image generation, data augmentation, reconstruction, and super-resolution.The survey links these uses to limited medical-image data and acquisition constraints.
  • Image synthesis and translation: GAN-based systems synthesize retinal, neuronal, skin-condition, and multimodal medical images through conditional or image-to-image translation.DermGAN maps semantic skin-condition descriptions to realistic images while retaining specified traits.
  • Data augmentation: Augmented datasets improved tissue-recognition accuracy for three GAN models compared with baseline models trained without augmentation.The survey reports an increase in recognition accuracy after adding generated images.
  • Data augmentation: fNIRS-GAN augmentation raised classification accuracy to 0.733 for SVM and 0.746 for neural networks, versus 0.4 without augmentation.The application targets fNIRS-brain-computer-interface classification.
  • Data augmentation: GAN-generated mRNA samples increased training data fivefold and improved cancer-detection F1 score by 39%.The synthetic samples were combined with the original dataset.
  • Super-resolution: ESRGAN produced the sharpest, highest-contrast OCT images qualitatively but performed worst on PSNR and SSIM.The result illustrates a mismatch between perceptual appearance and these quantitative metrics.

4.4 Biology

GANs are applied in biology to generate and optimize protein sequences, augment or impute biological data, predict disease-related molecular changes, and create microscopy images. The survey describes both sequence- and image-based biological generation.

  • Protein engineering: GANs generate valid functional protein sequences and optimize them for specified properties.ProteinGAN learns diverse functional sequences from complex multidimensional amino-acid sequence space.
  • Protein engineering: Antibody-GAN generates diverse single-chain and paired-chain antibody libraries and supports transfer learning for property-specific molecules.Reported target properties include MHC class II binding and complementarity-determining-region characteristics.
  • Protein engineering: FBGAN uses an analyzer feedback loop to replace low-scoring generated gene sequences with higher-scoring ones during training.An external differential neural-network analyzer assigns sequence scores.
  • Biological data: GANs address biological data scarcity through augmentation and imputation, including realistic single-cell RNA-seq generation and gene-expression completion.GGAN conditions target-gene expression on landmark-gene expression, while scIGAN learns nonlinear gene-gene correlations.
  • Biological data: WGAN+GP was applied to RNA-seq data from a 5xFAD mouse model to analyze Alzheimer’s-disease molecular progression.The model addressed bulk RNA-seq data with fewer gene-expression variations and fewer genes.
  • Biological imaging: In cell microscopy experiments, LSGAN was the most stable and produced higher-quality images than DCGAN and WGAN.GANs were also used for electron-microscope and fluorescence-microscopy image generation.

4.5 Astronomy

GANs are applied in astronomy for synthetic image generation, image translation, data augmentation, and spectral denoising. Reported applications include faster exoplanet-atmosphere retrieval and radio-survey flux recovery.

  • Astronomical GAN applications include image translation, data augmentation, and spectral denoising.
  • RadioGAN recovers extended flux density for nearly half of sources within a 20% margin of error.
  • GANs generate synthetic astronomical images, including Hubble-like XDF images, cosmological images, and realistic galaxy images.
  • ExoGAN was found to be up to 300 times faster than a standard retrieval for large spectral ranges.
  • ExoGAN works across a wide range of instruments and wavelength ranges without additional training.

4.6 Remote Sensing

Remote-sensing applications of GANs span image generation and augmentation, super resolution, pan-sharpening, dehazing, despeckling, and cloud removal. Reported methods target sharper, higher-resolution, and restored imagery under limited paired-data conditions.

  • Remote-sensing GAN applications include super resolution, pan-sharpening, data augmentation, haze removal, restoration, and cloud removal.
  • Data generation or augmentation: MARTA GAN generates 256×256 images, whereas DCGAN produces images with 64 × 64 resolution.
  • Super Resolution: HRPGAN preserves textures and sharp edges using a PatchGAN-inspired architecture without batch normalization.
  • Pan-Sharpening: PSGAN generates high-quality pan-sharpened images with fine spatial details and high-fidelity spectral information.
  • Pan-Sharpening: Experiments found two-stream architecture preferable to stacking, while batch normalization and self-attention were undesirable for pan-sharpening.
  • Haze removal and Restoration: ES-CCGAN performs unsupervised dehazing with unpaired images and uses edge-sharpening, cycle-consistency, and perceptual-consistency losses.

4.7 Material Science

GANs support material-science design by generating crystal structures, microstructures, inorganic materials, and metaporous designs. The surveyed methods incorporate composition, physical invariances, feature transfer, or simulation-based evaluation.

  • Microstructure design: Physics-aware GANs enforce explicit physical invariances alongside implicit restrictions derived from image data when recreating microstructures.
  • Crystal structure generation: CrystalGAN generates chemically stable crystallographic structures with enhanced domain complexity through two GAN stages and feature transfer.
  • Crystal structure generation: Composition-Conditioned Crystal GAN generates materials with desired chemical composition using a one-hot encoded composition vector.
  • Architectured materials: GANs generate complex architectured material designs that require no prior knowledge and can be applied across a wide range of applications.
  • Inorganic materials design: MatGAN learns implicit chemical compositional rules to generate hypothetical yet chemically sound inorganic materials.
  • Topological design: GAN-generated metaporous designs achieved satisfactory broadband absorption performance confirmed by FEM simulation and experimental evaluation.

4.8 Finance

Financial time series are difficult to model because of complex statistical properties, temporal dependence, and limited labeled historical data. GAN applications address synthetic-data generation, forecasting, stylized financial behavior, and manipulation detection.

  • Financial data modeling involves complex statistical properties, dynamic stochastic factors, temporal correlation, and limited labeled historical data.
  • GANs generate realistic synthetic financial time-series data for finance applications.
  • Financial time-series generation: Quant GANs capture long-range dependence, including volatility clusters, in S&P 500 index data.
  • Financial time-series generation: FIN-GAN captures stylized facts of price returns, including fat-tailed distributions, volatility clustering, leverage effects, and gain/loss asymmetry.
  • Market manipulation detection: A trading-data detection system achieved 68.1% accuracy in detecting pump-and-dump manipulations in unseen data.

4.9 Marketing

GANs are surveyed as tools for generating and editing marketing, logo, and fashion-design imagery. The reviewed methods use conditioning, staged refinement, clustering, and attribute-focused discrimination to produce controllable designs and apparel images.

  • Marketing: GANs can synthesize novel marketing designs and replace real people in advertisements with synthetic images or videos to address privacy concerns.
  • Marketing: iWGAN generates many logo variations from parameters such as shape and colour to facilitate and expedite logo design.
  • Marketing: A clustered GAN trains on multimodal unlabeled data to stabilize training, prevent mode collapse, and achieve higher-quality logo samples.
  • Marketing: LoGAN generates logos conditioned on twelve predetermined colors using a classifier network and WGAN-GP loss for improved training stability.
  • Fashion Design: Fashion GANs map clothing attributes to outfit pairs, edit garments according to user-defined attributes, and use specialized discriminators for local structure and attribute correctness.
  • Fashion Design: Design-AttGAN modifies AttGAN to avoid conflict between attribute-classification and reconstruction losses during garment-image editing.

4.11 Sports

GANs support sports-data augmentation, prediction, simulation, and text generation, while also enabling music generation across symbolic, audio, and variable-length formats. The reviewed applications report improved classification, high-fidelity audio, and music outputs comparable to human compositions.

  • Sports: GANs are used to generate sports text, augment datasets, predict activity, and simulate sports actions despite limited labeled data.
  • Sports: WGAN-GP generates sports-news sentences from game scores, demonstrating a potential GAN application in natural-language processing.
  • Sports: JointsGAN increased dribbling-style classification accuracy from 88.14% to 89.83% by conditioning GAN-based video augmentation on player joints.
  • Sports: MSS-GAN combines visual embeddings with episodic and semantic memory to predict tennis shot type and location through a response-generation network.
  • Sports: BasketballGAN generates set plays from a coach-sketched offensive tactic and random noise while optimizing Wasserstein, dribbler, defender, passing, and acceleration losses.
  • Music: MuseGAN and related models generate symbolic multi-track music, including piano rolls with fewer excessively fragmented notes than existing approaches.
  • Music: GANSYNTH generates high-fidelity, locally coherent audio and outperforms WaveNet while producing samples much faster.
  • Music: INCO-GAN synthesizes complete variable-length music whose frequency vectors reach cosine similarity of up to 0.987 with human-composed music.

5 LIMITATIONS OF GANS AND FUTURE DIRECTION

GAN training remains difficult because models may collapse to limited outputs, fail to converge, or provide vanishing gradients. The review discusses mitigation strategies and future work aimed at improving diversity, stability, theory, and efficiency.

  • Limitations: Mode collapse occurs when a Generator produces only one type or a small set of outputs while seeking an easily convincing sample.
  • Mitigation strategies: Wasserstein distance and unrolled GANs are proposed to reduce mode collapse by improving distribution comparison or limiting over-optimization for one Discriminator.
  • Limitations: GAN training may not converge because maintaining balance between Generator and Discriminator networks is difficult, and gradient descent guarantees Nash equilibrium only for convex functions.
  • Mitigation strategies: Regularization methods such as adding noise to Discriminator inputs and penalizing Discriminator weights are used to improve convergence.
  • Limitations: Vanishing gradients arise when an overly accurate Discriminator produces near-zero gradients, slowing or stopping Generator learning.
  • Mitigation strategies: Loss modifications can alleviate vanishing gradients but may leave unstable, oscillating training, while Wasserstein loss is designed to prevent them at Discriminator optimality.
  • Future directions: Future research should improve image quality without losing diversity, strengthen theoretical analysis, stabilize and accelerate training, and combine complementary techniques.
  • Future directions: Addressing individual training problems can involve trade-offs between desired output quality and training stability, motivating holistic approaches.

6 CONCLUSION

The review presents state-of-the-art GAN models, applications across diverse domains, and open research topics. It aims to help researchers understand GANs and assess their potential for specific tasks.

  • The paper surveys state-of-the-art GAN models and their applications across a wide variety of domains.
  • The review covers GAN theory, applications, and open research topics while noting GANs’ use of unlabeled data.
  • The study is intended to help academic and industry researchers understand GANs and evaluate potential applications for specific tasks.
Loading 2110.01442v1…