Source-linked AI summary
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Ming-Yu Liu, Xun Huang, Jiahui Yu, Ting-Chun Wang, Arun Mallya
TL;DR
GANs address the challenge of defining objectives for visual generation while supporting unconditional and conditional synthesis. This paper surveys GAN architectures, stabilization strategies, and applications across image and video synthesis, reporting broad use in visual processing and neural rendering. It also identifies persistent training instability and evaluation-metric shortcomings.
Problem
Visual generation lacks an obvious hand-designed objective, while GAN training is difficult because generator and discriminator updates continually change the learning target.
Method
The paper provides a comprehensive overview of GAN architectures, training-stabilization strategies, and applications in image translation, image processing, video synthesis, and neural rendering.
Results
GANs have been applied to photorealistic image restoration, super-resolution, compression, 3D-aware rendering, and video synthesis, with reported improvements in visual quality across several tasks.
Takeaways & Limitations
GANs provide a flexible data-driven framework for visual synthesis and have become a building block for diverse content-creation and rendering systems.
Takeaways & Limitations
GANs remain more unstable to train than supervised discriminative or likelihood-based generative models, and commonly used evaluation metrics have important shortcomings.
Abstract
from arXiv · showhide
The generative adversarial network (GAN) framework has emerged as a powerful tool for various image and video synthesis tasks, allowing the synthesis of visual content in an unconditional or input-conditional manner. It has enabled the generation of high-resolution photorealistic images and videos, a task that was challenging or impossible with prior methods. It has also led to the creation of many new applications in content creation. In this paper, we provide an overview of GANs with a special focus on algorithms and applications for visual synthesis. We cover several important techniques to stabilize GAN training, which has a reputation for being notoriously difficult. We also discuss its applications to image translation, image processing, video synthesis, and neural rendering.
1 INTRODUCTION
GANs replace hand-designed generation objectives with data-derived objectives learned through a generator–discriminator game. The framework supports unconditional and controllable conditional synthesis, while training remains difficult because each network continually changes the other’s target.
- Core framework: GANs jointly train a generator and discriminator in a zero-sum game so generated-data distributions can approach real-data distributions.The generator synthesizes fake data, while the discriminator distinguishes real from fake data.
- Motivation and objective: GANs derive generation objectives from training data and discriminator feedback rather than hand-designing a function for visual properties such as “felineness.”The discriminator’s classification task guides generator updates using current network weights.
- Conditional synthesis: Conditional GANs add an input control signal such as an image, text, or categorical label, unlike unconditional GANs that map noise to images.The discriminator also uses the conditioning information when distinguishing real from fake.
- Training challenges: GAN training is notoriously difficult because generator updates change the fake-data distribution, creating a new classification problem for the discriminator.An overly strong discriminator can produce gradients that push the generator toward numerical instability.
- Applications: GANs support applications including semantic image synthesis, image-to-image translation, restoration, super-resolution, inpainting, video synthesis, and neural rendering.The survey organizes these applications across image translation, image processing, video synthesis, and neural rendering.
2 RELATED WORKS
The paper situates GANs among deep generative models with different graph structures, inference procedures, and modeling constraints. Compared with alternatives, GANs are discussed alongside models that trade off likelihood evaluation, sampling efficiency, stability, output sharpness, or architectural flexibility.
- Model families: The related-work comparison covers deep Boltzmann machines, variational autoencoders, autoregressive models, and normalizing flow models alongside GANs.These models represent different approaches to modeling data-generation processes.
- Deep Boltzmann machines: Deep Boltzmann machines use undirected graphs and MCMC sampling, which makes training and generation scale poorly when efficient sampling is difficult.Their probability model uses an energy function and partition function.
- Variational autoencoders: VAEs use inference and generation networks with latent variables, enabling efficient sampling but tending to produce blurry output images.Training maximizes an evidence lower bound, and generation samples from a Gaussian prior.
- Autoregressive models: Deep autoregressive models generate variables sequentially under an assumed ordering, which is unnatural for image grids because images lack a temporal ordering.Their conditional factorization generates each portion from variables generated or observed earlier.
- Normalizing flows: Normalizing flows transform simple distributions through invertible, differentiable mappings and support direct density evaluation, but invertibility limits applicable architectures.The architectural constraint applies across the stacked transformations.
3 LEARNING
GAN learning formulates generation as a minimax game with multiple loss, optimization, regularization, and architecture choices. These choices address training instability, improve gradient behavior, and support controllable generation.
- GAN objectives: GAN training uses a minimax objective in which generator and discriminator parameters are optimized through a utility function.The discriminator and generator objectives are expressed through separate output transformations for real and fake images.
- GAN objectives: Common GAN losses measure discrepancies between true and generated distributions using divergences or integral probability metrics.Examples include saturated, non-saturated, Wasserstein, least-square, and hinge losses.
- Optimization: Alternating updates are generally more stable than simultaneous updates because generator updates use the newly updated discriminator parameters.Simultaneous updates can share computation more efficiently, whereas alternating updates reflect the latest discriminator state.
- Optimization: ADAM is widely used for GAN training, and TTUR commonly sets the discriminator learning rate 2 to 4 times higher than the generator rate.The generator learning rate is usually set to 0.0001 in the described setting.
- Regularization: Regularization methods such as gradient penalties, feature matching, perceptual loss, model averaging, and spectral normalization target instability and improve generated-image quality.GP penalizes deviations in gradient norms, while spectral normalization divides each discriminator weight matrix by its largest eigenvalue.
- Network architecture: GAN architectures evolved from MLPs to convolutional and residual generators, with conditional normalization and convolutions adding control over generation outputs.Conditional discriminators use auxiliary classifiers, input concatenation, or projection-based image–label embeddings.
4 IMAGE TRANSLATION
GAN-based image translation maps images between domains in supervised or unsupervised settings, with methods addressing controllability, multimodal outputs, limited data, and semantic ambiguity.
- Image-to-image translation maps an image from one domain to a corresponding image in another, using paired or unpaired training data.
- Supervised Image Translation: Pix2pix combines conditional GANs with pixel-wise ℓ1 loss and patch-wise discrimination for supervised translation.
- Supervised Image Translation: Pix2pix is restricted to one-to-one mappings, motivating methods that encourage generators to represent multimodal output variations.
- Supervised Image Translation: SPADE improves translation quality with spatially-adaptive normalization and supports style-image control over the output appearance.
- Unsupervised Image Translation: Unsupervised translation methods address unavailable paired data through cycle consistency, shared latent spaces, distance preservation, or single-model multi-domain translation.
- Unsupervised Image Translation: MUNIT enables diverse translations by separating domain-invariant content from domain-specific style, while other methods use spatial attention or style guidance to control outputs.
- Unsupervised Image Translation: Existing unsupervised methods generally still require many unpaired images, although some methods target settings with few or test-time target examples.
- Unsupervised Image Translation: Unsupervised translation remains ill-posed because infinitely many mappings satisfy its constraints, making generator architectural inductive bias important for meaningful results.
5 IMAGE PROCESSING
GANs improve perceptual quality across image restoration and enhancement tasks, including super-resolution, compression, deblurring, and inpainting, while introducing task-specific limitations.
- GANs are applied to restoration and enhancement tasks where perceptual quality is a primary evaluation criterion, including super-resolution, denoising, deblurring, compression-artifact removal, and inpainting.
- Image Restoration and Enhancement: Restoration quality is evaluated using distortion metrics such as MSE, PSNR, and SSIM alongside perceptual metrics such as NR.
- Image Restoration and Enhancement: ESRGAN revises SRGAN’s architecture and losses, producing more realistic textures and winning the 2018 PIRM challenge.
- Image Restoration and Enhancement: Super-resolution methods trained on synthetically downsampled pairs may not directly handle sensor-captured low-resolution images, motivating unsupervised approaches.
- Image Restoration and Enhancement: GAN-based compression methods target realistic detail at low bit rates, with Agustsson et al. reporting performance that outperforms other codecs in that setting.
- Image Inpainting: Deep learning and GAN-based inpainting methods are more scalable than PatchMatch and can leverage large-scale data, but reconstructing foreground details remains difficult.
- Image Inpainting: Inpainting methods can handle free-form masks, while edge-guided approaches decompose the task into foreground-edge prediction followed by final-image synthesis.
6 VIDEO SYNTHESIS
The section surveys GAN-based video synthesis, emphasizing temporal consistency, conditional generation, face reenactment, and pose transfer. It contrasts subject-specific and subject-agnostic methods across images and videos.
- Video synthesis must preserve temporal consistency, using techniques such as temporal discriminators, flow-warping losses, input smoothing, or post-processing.
- Conditional video synthesis includes future-frame prediction and video-to-video generation conditioned on input content.
- 6.1 Face Reenactment: Face reenactment transfers expressions and head poses, unlike face swapping, which pastes a subject’s face region onto another subject.
- 6.1 Face Reenactment: Subject-specific reenactment models are limited to predetermined identities, whereas subject-agnostic methods target general identities but historically struggled with spontaneous head motion.
- 6.1 Face Reenactment: Recent subject-agnostic frameworks use a single 2D target image and facial landmarks or learned keypoints to synthesize talking videos driven by arbitrary motions.
- 6.2 Pose Transfer: Pose transfer extends face reenactment to whole bodies, where varied poses make synthesis harder; recent methods generate high-quality, high-resolution videos but may remain person-specific.
7 NEURAL RENDERING
The section presents neural rendering as a combination of classical rendering and generative modeling, organized around 2D enhancement or native 3D feature processing. GAN losses improve visual realism, while 3D-aware representations support pose control and consistent views.
- Neural rendering combines classical rendering with generative models to produce flexible visual outputs from incomplete or difficult-to-specify world descriptions.
- GAN-based neural rendering separates 2D postprocessing methods from approaches incorporating differentiable 3D↔2D operations and implicit geometry.
- 7.1 3D to 2D projection as a preprocessing step: 2D-domain pipelines project 3D worlds into feature maps containing information such as color, depth, normals, or segmentation before generating realistic RGB images.
- 7.1 3D to 2D projection as a preprocessing step: Adversarial loss improves conditioned rendering outputs, including more detailed and plausible colors when reconstructing images from sparse 3D point attributes.
- 7.2 3D ↔2D transform as a part of network training: Native 3D operations let networks model scene geometry and appearance in feature space, enabling 3D reasoning, pose control, and view consistency.
- 7.2 3D ↔2D transform as a part of network training: 3D-aware models use varied representations and supervision regimes, while GAN losses help capture realism and guide upstream features toward real-image distributions.
8 LIMITATIONS AND OPEN PROBLEMS
The paper identifies unresolved problems in evaluating GANs, stabilizing their training, interpreting their internal representations, and detecting generated content. These limitations affect both scientific comparison and safe deployment.
- Evaluation metrics: Common metrics have important weaknesses: Inception Score misses intra-class mode collapse, FID has a biased estimator, and KID can have high variance.
- Instability: GAN training remains more unstable than supervised or likelihood-based models, with eventual collapse reported even for BigGAN on ImageNet.
- Instability: GAN performance is generally sensitive to hyper-parameter choices.
- Interpretability: Despite high image quality, how GAN generators internally represent image structure remains poorly understood.
- Forensics: GAN-generated content raises fraud and fake-news concerns, making detection important as generated images may become harder to distinguish.
9 CONCLUSION
The paper provides a comprehensive overview of GAN algorithms and visual-synthesis applications, including image translation, image processing, video synthesis, and neural rendering. It also identifies open problems to motivate future research.
- The paper surveys GAN architectures, training-stabilization strategies, and applications in image translation, image processing, video synthesis, and neural rendering.
- The authors identify open problems intended to inspire future research on GANs.