Source-linked AI summary
On the Effectiveness of Least Squares Generative Adversarial Networks
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, Stephen Paul Smolley
TL;DR
Regular GAN losses can cause vanishing gradients during generator training. The paper introduces LSGANs, which use least-squares losses and an objective tied to Pearson χ2 divergence, and reports higher image quality and more stable training than regular GANs, with LSGANs-GP training all tested difficult architectures.
Problem
Regular GAN sigmoid cross-entropy losses may cause vanishing gradients during learning, while GAN image quality and training stability remain challenging.
Method
LSGANs apply least-squares losses to both discriminator and generator, with an objective that minimizes Pearson χ2 divergence.
Results
LSGANs generate higher-quality images and train more stably than regular GANs; LSGANs-GP succeeds on all six difficult architectures, including 101-layer ResNet.
Takeaways & Limitations
The derived Pearson χ2 objective outperforms classical least-squares classification, while LSGANs-GP shows comparable stability to WGANs-GP.
Takeaways & Limitations
Experiments compare LSGANs with NS-GANs because NS-GANs perform much better than M-GANs.
Abstract
from arXiv · showhide
Unsupervised learning with generative adversarial networks (GANs) has proven to be hugely successful. Regular GANs hypothesize the discriminator as a classifier with the sigmoid cross entropy loss function. However, we found that this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we propose in this paper the Least Squares Generative Adversarial Networks (LSGANs) which adopt the least squares loss for both the discriminator and the generator. We show that minimizing the objective function of LSGAN yields minimizing the Pearson $χ^2$ divergence. We also show that the derived objective function that yields minimizing the Pearson $χ^2$ divergence performs better than the classical one of using least squares for classification. There are two benefits of LSGANs over regular GANs. First, LSGANs are able to generate higher quality images than regular GANs. Second, LSGANs perform more stably during the learning process. For evaluating the image quality, we conduct both qualitative and quantitative experiments, and the experimental results show that LSGANs can generate higher quality images than regular GANs. Furthermore, we evaluate the stability of LSGANs in two groups. One is to compare between LSGANs and regular GANs without gradient penalty. We conduct three experiments, including Gaussian mixture distribution, difficult architectures, and a newly proposed method --- datasets with small variability, to illustrate the stability of LSGANs. The other one is to compare between LSGANs with gradient penalty (LSGANs-GP) and WGANs with gradient penalty (WGANs-GP). The experimental results show that LSGANs-GP succeed in training for all the difficult architectures used in WGANs-GP, including 101-layer ResNet.
1 INTRODUCTION
The paper proposes LSGANs to address vanishing gradients in regular GANs by using least-squares losses, and evaluates their image quality and training stability against established GAN objectives.
- Regular GAN generator losses can produce vanishing gradients for fake samples, making generator updates difficult.
- Least-squares loss penalizes fake samples far from the decision boundary, generating gradients that pull them toward the boundary and closer to real data.
- LSGANs use least-squares losses for both discriminator and generator, with the objective linked to minimizing Pearson χ2 divergence.
- LSGANs generate higher-quality images than NS-GANs in qualitative evaluations and outperform NS-GANs on FID across four datasets.
- LSGANs perform more stably than NS-GANs in experiments involving Gaussian mixtures, difficult architectures, and low-variability datasets.
- LSGANs-GP successfully train all six difficult WGANs-GP architectures, including a 101-layer ResNet, while gradient penalty adds computational and memory costs.
2 RELATED WORK
Prior work established GANs as effective unsupervised generative models while identifying persistent challenges in image quality and training stability. Researchers addressed these challenges through architectural changes, regularization, alternative objectives, and divergence analysis.
- GANs avoid approximate inference and can be trained end-to-end through differentiable networks for unsupervised tasks.
- Prior methods improved generated-image quality through architectures such as DCGANs and Laplacian pyramid GANs.
- Stability research explored objective functions, energy-based models, unrolled optimization, reconstruction regularization, and alternative divergences.
- Wasserstein distance and gradient penalties were investigated to improve stability, with different penalties applied around real data or between real and fake samples.
3 METHOD
LSGANs replace sigmoid cross-entropy with least-squares losses to address vanishing gradients and improve generation. Their objective can minimize Pearson χ2 divergence under specific label conditions, while an alternative least-squares classification objective performs worse in FID and convergence speed.
- LSGAN objective: LSGANs use least-squares losses for the discriminator and generator instead of the regular GAN losses.The generator target assigns the discriminator's desired output to fake samples.
- Motivation: Regular GAN minimax and non-saturating losses can produce vanishing gradients for some fake samples.The non-saturating loss saturates for relatively large inputs, while the minimax loss saturates for relatively small inputs.
- Motivation: Least-squares loss penalizes correctly classified fake samples according to their distance from the decision boundary, directing generation toward that boundary.This penalty provides gradients for samples that would otherwise receive almost no gradient.
- Relation to χ2 divergence: For b − c = 1 and b − a = 2, optimizing the LSGAN objective minimizes the Pearson χ2 divergence between pd + pg and 2pg.The result follows from the optimal discriminator for a fixed generator.
- Properties of χ2 divergence: The Pearson χ2 objective has no zero-forcing property, making LSGANs less mode-seeking and potentially alleviating mode collapse.This contrasts with the mode-seeking behavior associated with KL(pg∥pd).
- Parameter selection: The derived Pearson χ2 objective shows better FID results and faster convergence than the classical least-squares classification objective.The derived objective is Eq. (13), while the classical classification objective is Eq. (14).
4 EXPERIMENTS
The experiments compare LSGANs with NS-GANs for image quality and stability, and compare gradient-penalized LSGANs with WGANs-GP.
- Experimental design: Image-quality experiments compare LSGANs with NS-GANs, using NS-GANs as the baseline method.The comparison uses the same network architecture for the relevant baseline experiments.
- Stability evaluation: Stability without gradient penalty is evaluated through three experiments comparing LSGANs with NS-GANs.The experiments cover Gaussian mixture distributions, difficult architectures, and datasets with small variability.
- Stability evaluation: Stability with gradient penalty is evaluated by comparing LSGANs-GP with WGANs-GP.This forms the second experimental group.
4.1 Implementation Details
The implementation builds on a public TensorFlow DCGAN implementation and uses specified optimization, initialization, normalization, and image-resolution settings.
- Implementation: The implementation is based on a public DCGAN implementation using TensorFlow.The paper also states that the implementation is publicly available.
- Optimization settings: The learning rate is 0.0002, except for LSUN-scenes where it is 0.001, and the mini-batch size is 64.These are the stated optimization settings.
- Preprocessing and initialization: Images are scaled to [-1,1] for a generator using Tanh, while Adam uses β1 = 0.5 and variables use Gaussian initialization with mean 0 and standard deviation 0.02.The initialization and preprocessing settings follow the implementation description.
4.2 Image Quality
LSGANs produce sharper, more detailed images than NS-GANs in qualitative comparisons, while FID and human judgments provide quantitative support across several datasets.
- Qualitative Evaluation: Smooth latent-space interpolations indicate that LSGANs learn semantic representations rather than merely memorizing training examples.The interpolation procedure varies between two randomly sampled noise vectors.
- Qualitative Evaluation: LSGANs generate sharper and more exquisite scene and cat images than NS-GANs, with fewer artificial artifacts in cat hair.The comparisons use the same network architecture for the models.
- Quantitative Evaluation: FID measures feature-space distance between generated and real images, with smaller values indicating closer distributions.The metric approximates Inception features as multidimensional Gaussian distributions and is reported as more consistent with human judgment than inception score.
- Quantitative Evaluation: LSGANs(−110) outperform NS-GANs on all four evaluated datasets and outperform WGANs-GP on three datasets.LSGANs(−110) also outperform the classical LSGANs(011) objective on all four datasets.
- Quantitative Evaluation: LSGANs reach a relatively optimal FID around 22 in 1,100 minutes, compared with 4,600 minutes for WGANs-GP.The learning-process comparison aligns LSGANs and NS-GANs by iterations, but aligns LSGANs and WGANs-GP by wall-clock time.
- Quantitative Evaluation: In a 4,000-vote bedroom-image study, LSGANs receive 56.4% of votes versus 43.6% for NS-GANs.The authors report this as a 12.8% increase in votes over NS-GANs.
4.3 Training Stability
The paper evaluates LSGAN stability against NS-GANs without gradient penalty and against WGANs-GP with gradient penalty across distributions, architectures, and low-variability datasets. LSGANs show fewer mode-collapse failures and successful training across the difficult architectures tested.
- Gaussian Mixture Distribution: NS-GANs produced one or two modes in 99 of 100 runs, compared with 5 runs for LSGANs.
- Difficult Architectures: Without batch normalization, LSGANs sometimes learned relatively good images with Adam, whereas NS-GANs never learned successfully in the reported BNG setting.The experiment used BNG and BNGD architectures with Adam and RMSProp on LSUN-bedroom.
- Datasets with Small Variability: On MNIST and two synthetic low-variability digit datasets, NS-GANs succeeded only on MNIST, while LSGANs succeeded on all three datasets.
- Datasets with Small Variability: Increasing dataset variability improved LSGAN image quality and relieved the mode-collapse problem in the synthetic digit experiment.The second synthetic dataset added random rotation to horizontal shifts and had larger variability than the first.
- Evaluation with Gradient Penalty: LSGANs-GP trained successfully on all six difficult architectures evaluated for WGANs-GP, including the 101-layer ResNet.The comparison used gradient penalty for both models, with WGANs-GP images duplicated from prior work.
4.4 Comparison of Two Parameter Schemes
The paper compares two LSGAN parameter schemes on SVHN and observes faster convergence for LSGANs(−110), with similar results on LSUN-bedroom and the cat dataset.
- LSGANs(−110) converged faster than LSGANs(011) on SVHN, with similar results on LSUN-bedroom and the cat dataset.
4.5 Suggestions in Practice
The paper recommends using LSGANs(−110) without gradient penalty when possible and saving generated images throughout training because image quality can fluctuate.
- Use LSGANs(−110) without gradient penalty when it works, because gradient penalty adds computational and memory costs.
- Record generated images every hundred or thousand iterations because LSGAN image quality may shift between good and bad during training.
5 CONCLUSIONS AND FUTURE WORK
The paper proposes LSGANs to address vanishing gradients and reports higher-quality images, better performance for the derived Pearson χ2 objective, and improved stability across its experiments.
- LSGANs generate higher-quality images than regular GANs and perform better with the derived Pearson χ2 objective than with least-squares classification.
- The experiments compare LSGAN stability with regular GANs without gradient penalty and with WGANs-GP using gradient penalty.