Source-linked AI summary
Least Squares Generative Adversarial Networks
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, Stephen Paul Smolley
TL;DR
Regular GANs can suffer vanishing gradients and unstable learning, limiting unsupervised generative modeling. This paper introduces LSGANs, which replace the discriminator’s sigmoid cross-entropy loss with least squares, and reports higher-quality images and more stable training than regular GANs.
Problem
Regular GANs’ objective can cause vanishing gradients and unstable learning, while unsupervised generative models face effectiveness limitations.
Method
LSGANs replace the discriminator’s sigmoid cross-entropy loss with a least-squares objective that minimizes Pearson χ^2 divergence.
Results
LSGANs generated higher-quality images and trained more stably than regular GANs across scene, Gaussian-mixture, and handwritten-character experiments.
Takeaways & Limitations
LSGANs provide a more stable alternative to regular GANs for the evaluated image-generation tasks.
Takeaways & Limitations
The paper leaves pulling generated samples directly toward real data as future work.
Abstract
from arXiv · showhide
Unsupervised learning with generative adversarial networks (GANs) has proven hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss function for the discriminator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson $χ^2$ divergence. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stable during the learning process. We evaluate LSGANs on five scene datasets and the experimental results show that the images generated by LSGANs are of better quality than the ones generated by regular GANs. We also conduct two comparison experiments between LSGANs and regular GANs to illustrate the stability of LSGANs.
1 Introduction
The paper introduces LSGANs, which replace the discriminator’s sigmoid cross-entropy loss with least-squares loss to address vanishing gradients and improve generated-image quality and training stability. It connects the LSGAN objective to Pearson χ2 divergence and evaluates the approach across scene and handwritten-character datasets.
- Introduction: Regular GANs’ sigmoid cross-entropy loss can produce vanishing gradients when fake samples lie on the correct side of the decision boundary but remain far from real data.This limits generator updates and generated-image quality in some realistic tasks.
- Introduction: LSGANs relieve vanishing gradients by penalizing samples according to their distance from the decision boundary, generating more gradients for generator updates.The paper identifies the objective function as a partial cause of GAN-learning instability.
- Introduction: LSGANs adopt a least-squares discriminator loss, whose objective minimizes the Pearson χ2 divergence.This is the paper’s central methodological contribution.
- Introduction: Experiments report that LSGANs generate more realistic images than regular GANs and exhibit more stable learning.The paper conducts numerous comparison experiments to assess stability.
- Introduction: Two LSGAN architectures target 112 × 112 image generation and tasks with many classes, including a handwritten Chinese character dataset with 3470 classes.The first architecture is evaluated on scene datasets, while the second generates readable characters.
2 Related Work
Related work spans deep generative models based on RBMs, the emergence and applications of GANs, and efforts to improve GAN image quality and training stability. These efforts include architectural changes, intermediate-layer mean-square objectives, and analyses or modifications of GAN objectives.
- Deep generative models: RBMs underpin several deep generative models, including DBNs and DBMs, and have modeled image and document distributions.DBNs have been especially successful for image classification through feature representation extraction.
- Generative adversarial networks: GANs introduced game-theoretic learning without requiring approximation methods and, like VAEs, use differentiable networks for unsupervised tasks.Applications include image generation, image super-resolution, and text-to-image tasks.
- Image quality: Because generated-image quality remains challenging, prior work proposed DCGANs with convolutional layers and LAPGANs based on a Laplacian pyramid framework.These are described as efforts to improve GAN image quality.
- Image quality: One approach seeks convergence by matching generated samples to real-data statistics through mean square error on an intermediate discriminator layer.The objective operates on an intermediate layer rather than directly on the final discriminator output.
- Learning stability: GAN learning stability has motivated objective-function analyses, energy-based auto-encoders, and unrolled objectives designed to improve generator–discriminator balance.These works address stability as a critical issue in GAN training.
3 Method
LSGANs replace the sigmoid cross-entropy discriminator loss with a least-squares loss to address vanishing gradients and improve training stability. Their objective is connected to minimizing the Pearson χ2 divergence under specific coding conditions, with alternative coding schemes and two model architectures described.
- 3.2 LSGANs: LSGANs replace regular GANs’ sigmoid cross-entropy loss because it can cause vanishing gradients for correctly classified samples far from real data.The least-squares formulation is proposed to remedy this problem.
- 3.2 LSGANs: Penalizing correctly classified samples far from the decision boundary produces more generator gradients, relieves vanishing gradients, and supports more stable learning.The least-squares loss is flat only at one point, whereas sigmoid cross entropy saturates when x is relatively large.
- 3.2 LSGANs: When b − c = 1 and b − a = 2, minimizing the LSGAN objective minimizes the Pearson χ2 divergence between pd + pg and 2pg.The paper also gives a = −1, b = 1, and c = 0 as one example satisfying these conditions.
- 3.2 LSGANs: A second coding strategy sets c = b to make the generator produce samples as real as possible, and either resulting objective can be selected in practice.The authors observe similar performance for Equation 8 and Equation 9 and use Equation 9 to train the models.
- 3.3 Model architectures: The paper introduces a VGG-motivated architecture with added stride=1 deconvolutional layers and a second model for tasks with many classes, such as Chinese characters.The first model uses ReLU in the generator and LeakyReLU in the discriminator; the second addresses difficulties generating readable outputs across multiple classes.
4 Experiments
Experiments evaluate LSGANs on five LSUN scene datasets, compare their stability with regular GANs, and test conditional generation on 3,740-class handwritten Chinese characters. LSGANs produce higher-quality scene images, avoid the observed mode collapse of regular GANs, and generate readable characters with recoverable labels.
- Scene datasets: On five LSUN scene datasets, LSGANs generated better-quality images than DCGANs and EBGANs.The datasets were bedroom, kitchen, church, dining room, and conference room.
- Stability comparisons: Stability comparisons used BNG and BNGD architectures with Adam and RMSProp on the LSUN bedroom dataset.BNG excludes generator batch normalization, while BNGD excludes batch normalization from both generator and discriminator.
- Stability comparisons: Regular GANs began suffering mode collapse at step 15k on a 2D mixture of 8 Gaussian distributions, generating samples around a single valid mode.The experiment used simple generators and discriminators with three fully connected layers each.
- Handwritten Chinese characters: A conditional LSGAN successfully generated readable handwritten Chinese characters from a dataset containing 3740 classes.The generated images’ correct labels could be obtained through label vectors for applications such as data augmentation.
5 Conclusions and Future Work
The paper proposes LSGANs with two model architectures and reports higher-quality images than regular GANs for the first architecture’s scene-dataset evaluation. Future work includes pulling generated samples toward real data directly rather than toward the decision boundary.
- Conclusions: The paper proposes Least Squares Generative Adversarial Networks (LSGANs) and designs two model architectures.The architectures are shown in Figures 3 and 4.
- Conclusions: The first architecture is evaluated on several scene datasets and generates higher-quality images than regular GANs.
- Future Work: Future work will investigate pulling generated samples toward real data directly instead of toward the decision boundary.