PaperlayerView exact revision on arXiv

Source-linked AI summary

Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Guo-Jun Qi

arXiv:1701.06264v6cs.CV

TL;DR

Classic GANs lack a prior that controls the data distribution and may therefore require an infinite-capacity discriminator for theoretical consistency. This paper introduces LS-GAN, which learns margin-based losses under Lipschitz regularity, extends the framework to GLS-GAN and CLS-GAN, and reports competitive generation and classification performance. The analysis also identifies a trade-off between classification accuracy and image-generation quality in the conditional model.

  • Problem

    Classic GANs impose no prior on the data distribution and rely on an infinite-capacity non-parametric discriminator for distributional consistency.

  • Method

    LS-GAN learns a loss that separates real and generated samples by margins, while alternately training the generator to minimize generated-sample losses under Lipschitz regularity.

  • Results

    LS-GAN is theoretically shown to achieve distributional consistency and generalizability, while GLS-GAN includes LS-GAN and WGAN as special cases.

  • Takeaways & Limitations

    CLS-GAN extends LS-GAN to conditional generation and classification, with experiments showing competitive performance on image generation and classification tasks.

  • Takeaways & Limitations

    The conditional model faces a trade-off between image-generation quality and classification accuracy controlled by λ.

Abstract

from arXiv · show

In this paper, we present the Lipschitz regularization theory and algorithms for a novel Loss-Sensitive Generative Adversarial Network (LS-GAN). Specifically, it trains a loss function to distinguish between real and fake samples by designated margins, while learning a generator alternately to produce realistic samples by minimizing their losses. The LS-GAN further regularizes its loss function with a Lipschitz regularity condition on the density of real data, yielding a regularized model that can better generalize to produce new data from a reasonable number of training examples than the classic GAN. We will further present a Generalized LS-GAN (GLS-GAN) and show it contains a large family of regularized GAN models, including both LS-GAN and Wasserstein GAN, as its special cases. Compared with the other GAN models, we will conduct experiments to show both LS-GAN and GLS-GAN exhibit competitive ability in generating new images in terms of the Minimum Reconstruction Error (MRE) assessed on a separate test set. We further extend the LS-GAN to a conditional form for supervised and semi-supervised learning problems, and demonstrate its outstanding performance on image classification tasks.

1 Introduction

The paper argues that classic GANs lack a suitable prior on data distributions, making their theoretical basis and generalization vulnerable. It proposes LS-GAN, Lipschitz regularization, and extensions aimed at distributional consistency and generalizable generation.

  • Classic GANs distinguish real from generated samples through an adversarial discriminator–generator game.
  • Classic GANs impose no prior on the data distribution and rely on an infinite-capacity non-parametric discriminator for distributional consistency.
  • An unregularized non-parametric discriminator may overfit the generator to training examples, weakening its ability to generate new samples.
  • LS-GAN trains a loss function with margins separating real and generated samples, while the generator minimizes generated-sample losses.
  • Lipschitz regularity on the data density supports LS-GAN distributional analysis and a PAC-style generalization result for reasonable training sets.
  • The generalized LS-GAN contains LS-GAN and WGAN as special cases, while its non-parametric analysis yields cone-shaped optimal-loss bounds with non-vanishing gradients.

2 Related Work

Related work develops GAN architectures, alternative training criteria, and generative approaches for image synthesis and classification. These methods vary in architecture and objective, while some theoretical analyses retain infinite-capacity assumptions.

  • GAN architectures use convolutional layers, Laplacian pyramids, and recurrent generative structures to synthesize images.
  • Energy-based GANs minimize an energy function with an autoencoder-structured discriminator, but their density-recovery analysis still assumes infinite discriminator capacity.
  • Other image-generation models match convolutional features, render 3D chair models, or iteratively construct images with recurrent variational autoencoders.
  • Generative representations have also been used to improve classification when labeled data are difficult or expensive to obtain.

3 Loss-Sensitive GAN

LS-GAN replaces the classic discriminator with a learned loss function that separates real and generated samples by margins. It alternates loss-function and generator optimization to seek an equilibrium.

  • The classic GAN trains a generator and discriminator adversarially until generated samples become indistinguishable from real samples.
  • LS-GAN learns a loss function whose real-sample loss is lower than a generated-sample loss by a desired margin.
  • The margin Δ(x, Gφ(z)) specifies the required loss separation between a real sample and its generated counterpart.
  • A nonnegative slack variable ξx,z relaxes constraint violations, with λ balancing expected real-sample loss against violation error.
  • With the loss fixed, the generator is optimized separately; the model alternates these objectives to seek an equilibrium.

4 Theoretical Analysis: Distributional Consistency

The theoretical analysis assumes compact support and Lipschitz continuity of the data density. Under this regularity condition, LS-GAN admits Lipschitz equilibria and generated density converges to the real density as λ grows.

  • As λ → +∞, the generated density PG*(x) converges to the real data density Pdata(x).
  • The analysis assumes that Pdata has compact support and is Lipschitz continuous with a bounded constant κ.
  • Lipschitz densities with compact support include a broad family of distributions, including densities associated with natural images.
  • Under the assumption, a Nash equilibrium exists where both the optimal loss Lθ* and generated density PG* are Lipschitz.
  • The Lipschitz condition replaces the classic GAN’s infinite-capacity discriminator assumption with a weaker regularity assumption for establishing distributional consistency.

5 Learning and Generalizability

The section replaces population expectations with empirical means and analyzes generalization for both the loss function and generator. It motivates Lipschitz regularization because sample complexity depends polynomially on model size and Lipschitz constants.

  • Learning from finite samples: Finite real examples and noise vectors approximate the population expectations in the LS-GAN objectives through empirical means.The empirical objectives use sampled real data, noise, and paired generated samples.
  • Learning from finite samples: The generalization analysis asks whether and how quickly the empirical objective difference |S_m − S| vanishes as the number of training examples increases.The analysis targets the loss-training objective directly rather than relying on an unrelated neural network distance.
  • Loss-function generalization: Under Lipschitz assumptions on the loss in θ and x, the sample complexity for bounding |S_m − S| is polynomial in model size N and log κ_L and κ.The assumptions bound the loss sensitivity to parameters and inputs, and the theorem holds with probability at least 1 − η.
  • Generator generalization: A parallel theorem establishes generator generalizability under Lipschitz assumptions in both generator parameters and noise inputs.The generator analysis uses parameter count M together with its Lipschitz constants.
  • Lipschitz regularization: The theory motivates bounding both κ and κ_L, but experiments omit the κ_L penalty because its sample-complexity effect is only log-linear and its gradient is computationally demanding.The omitted penalty requires computing a parameter gradient involving a second derivative.
  • Lipschitz regularization: Unlike methods that constrain the Lipschitz constant near one, LS-GAN directly minimizes the Lipschitz constant to target lower sample complexity.The paper states that these penalty choices arise from different theoretical perspectives and differ practically in experiments.

6 Wasserstein GAN and Generalized LS-GAN

This section relates LS-GAN and WGAN through Lipschitz regularity and introduces GLS-GAN as a common generalization. By varying its cost function, GLS-GAN includes both models and produces a broader family.

  • Comparison with Wasserstein GAN: LS-GAN and WGAN address classic GAN optimization problems from different perspectives, but both use Lipschitz regularity in training.The paper describes Lipschitz regularity as playing different roles in the two models.
  • Comparison with Wasserstein GAN: LS-GAN compares paired real and generated losses using a data-dependent margin, preventing unbounded-below minimization when the loss gap exceeds that margin.Pairing also prevents decomposition into separate first-order moments and coordinates the losses of real and generated samples.
  • Generalized LS-GAN: GLS-GAN replaces the hinge cost with any C(a) satisfying C(a) ≥ a and C(a) = a for a ≥ 0.These two properties are the only properties of the hinge cost used in the relevant objective proof.
  • Generalized LS-GAN: Varying the leaky-rectified-linear slope ν over (−∞, 1] yields a family of GLS-GANs beyond LS-GAN and WGAN.Other cost functions are also allowed when they satisfy the same two conditions.
  • Generalized LS-GAN: LS-GAN is GLS-GAN with ν = 0 and C_0(a) = (a)+, while WGAN is GLS-GAN with ν = 1 and C_1(a) = a.The WGAN equivalence identifies its critic f_w with −L_θ.
  • Experiments: The paper reports that GLS-GAN has competitive generalization performance for generating new images.This result is evaluated experimentally in the later image-generation section.

7 Non-Parametric Analysis

The non-parametric analysis characterizes optimal LS-GAN loss functions within a Lipschitz class. Their cone-shaped bounds and non-vanishing gradients explain why optimized losses can continue supplying generator updates, while direct non-parametric computation is impractical at scale.

  • Optimal loss functions: Theorem 4 identifies upper and lower optimal loss functions in F_κ that both minimize the empirical objective S_m.Any convex combination of these forms is also a global minimizer, so the global minimizer is not unique.
  • Gradient behavior: Both optimal bounds are cone-shaped and have non-vanishing gradients almost everywhere, with slopes of ±κ in the one-dimensional example.The cone shape is expressed through the distance function defining Lipschitz continuity.
  • Optimal loss functions: The upper and lower functions bound every optimal loss function in F_κ, and their values coincide on the sampled real and generated points.The bounds are supported in the convex hull of the sampled points.
  • Optimization formulation: The Lipschitz constant κ controls the size of the searched loss-function class and therefore the model’s available modeling ability.Larger κ permits a larger class of loss functions.
  • Optimization formulation: Although the non-parametric formulation is a linear program, its large number of constraints prevents direct use with very large training sets.The paper instead uses a parameterized network and iterative gradient descent for the loss and generator.
  • Gradient behavior: The linear objective and constraints of the non-parametric problem make saturation less likely than the logistic losses used by classic GANs.The paper connects this structure to avoiding vanishing gradients during generator training.

8 Conditional LS-GAN

Conditional LS-GAN generates samples under specified conditions and uses its learned conditional loss for supervised or semi-supervised classification. Its theory connects generated conditional densities to the true data density, while classification requires balancing generation against classification error.

  • Conditional generation: CLS-GAN conditions generation on y and defines Lθ(x, y) to measure misalignment between a sample and its condition.The generator produces Gφ(z, y), while real aligned examples are assigned lower losses than generated examples by a margin.
  • Conditional generation: The learned generator can produce samples whose conditional distribution follows Pdata(x|y), with convergence as λ → +∞.The result assumes each conditional density is Lipschitz and supported on a convex compact set.
  • Classification: The learned conditional loss can predict an example’s label, enabling CLS-GAN to support supervised classification and extension to semi-supervised learning.For unlabeled examples, the class minimizing Lθ(x, y = l) provides the best guessed label.
  • Classification: Training with labeled and generated examples can reveal class variations and provide an objective metric for evaluating learned feature representations.The classifier’s performance supplies a model-quality assessment based on classification behavior.
  • Classification: Classification requires choosing λ carefully because larger values improve the generative objective but can neglect the loss-minimization term important for reducing classification error.The paper therefore identifies a trade-off between generation and classification objectives.
  • Semi-supervised learning: The semi-supervised formulation introduces a separate loss for unlabeled examples and combines its objective with the labeled-data objective using γ.The unlabeled loss treats the minimum-loss class as the best guess while allowing an additional unknown-label probability.

9 Experiments

Experiments evaluate LS-GAN and CLS-GAN through image quality, gradient behavior, classification, and generalization metrics. LS-GAN remains competitive or more resilient than DCGAN, while GLS-GAN achieves the lowest reported test MRE among compared models.

  • Training Details: LS-GAN achieved competitive results using raw-image distances as loss margins, without external Inception or VGG networks.This supports robustness without requiring a sophisticated margin and agrees with the theory, which assumes no particular margin form.
  • Generated Images by LS-GAN: After 25 epochs on CelebA, LS-GAN and DCGAN produced images with no perceptible difference in visual quality.
  • Generated Images by LS-GAN: Removing generator batch normalization caused DCGAN to collapse, whereas LS-GAN continued generating faces without perceived deterioration or mode collapse.The experiment indicates greater resilience to this architectural change.
  • Generated Images by LS-GAN: LS-GAN’s generator gradient magnitude increased toward a common level across update frequencies, indicating a nonsaturated, approximately linear generator objective.The loss function was updated every iteration, while the generator was updated every 1, 3, or 5 iterations.

10 Conclusions

LS-GAN uses a margin-based loss function and Lipschitz regularity to support distributional consistency and generalization, while extending to conditional learning.

  • LS-GAN assigns lower losses to real samples than generated samples by a margin and trains the generator to minimize generated-sample losses.
  • Lipschitz regularity restricts the loss-function space, replacing the classic GAN’s need for a non-parametric discriminator with infinite modeling ability.
  • The analysis proves LS-GAN generalizability with a required number of training examples polynomial in loss-function complexity.
  • Figure 10 compares GAN reconstructions on CIFAR-10 using test-set Minimum Reconstruction Errors in parentheses.
  • Conditional LS-GAN reaches competitive performance on image generation and classification tasks.

A Proof of Lemma 2

The proof establishes that, as λ increases without bound, the optimal generated density converges to the real data density.

  • The proof assumes a Nash equilibrium for the LS-GAN optimization problem and uses loss-function comparisons at that equilibrium.
  • A sufficiently small coefficient α yields a nonexpansive loss function because the real and generated densities are Lipschitz.
  • Assuming the density inequality fails on a set of nonzero measure produces a contradiction with the equilibrium condition.
  • PG∗(x) converges to Pdata(x) as λ → +∞.
  • The WGAN-related argument likewise uses Lipschitz functions and a contradiction to establish its density relation.

C Proof of Theorem 2

Theorem 2 is proved by controlling empirical-to-population deviations across loss functions with concentration bounds, parameter nets, and union bounds.

  • The analysis initially disregards the first loss-minimization term because its role vanishes as λ approaches +∞.
  • McDiarmid’s inequality bounds empirical-to-population deviations for a single loss function using bounded margins and Lipschitz losses.
  • An ε-net covers the bounded loss-function parameter space, with cardinality controlled by the number of loss-function parameters.
  • The proof extends concentration from the net to all loss functions through parameter Lipschitzness and a union bound.
  • Generator updates create a sequence of generators requiring a union bound, but the iteration factor does not affect the lower bound on m because T is usually much smaller than N.

D Proof of Theorem 4 and Corollary 2

Theorem 4 and Corollary 2 characterize empirical minimizers through two Lipschitz extensions and establish corresponding lower and upper bounds.

  • The minimizer exists because Fκ is compact and the objective function is convex.
  • For each loss function in Fκ, the constructed functions bLθ and eLθ agree with it on all data points.
  • The equality eLθ(x(i)) = Lθ(x(i)) follows by combining Lipschitz-continuity bounds with the extension’s defining minimum.
  • Because the empirical objective depends only on sampled data-point values, a global minimum can be attained by functions of the form in (10).
  • Lipschitz-continuity arguments establish the lower and upper bounds for Lθ using bLθ(x) and eLθ(x).
Loading 1701.06264v6…