Source-linked AI summary
Geometric GAN
Jae Hyun Lim, Jong Chul Ye
TL;DR
GAN variants appear different despite sharing a common adversarial structure, while training can suffer from vanishing or unstable generator gradients. This paper exposes three geometric training steps, proposes an SVM maximal-margin formulation, and reports theoretical convergence and improved empirical behavior.
Problem
GAN training is difficult because accurate discriminators can produce vanishing generator gradients, while alternative updates may be unstable.
Method
The paper interprets GAN variants as three geometric operations and proposes geometric GAN using an SVM separating hyperplane with maximal margin.
Results
Geometric GAN outperforms existing GANs across all tested datasets, and its alternating optimization has a Nash-equilibrium convergence result.
Takeaways & Limitations
The geometric view provides a common interpretation of GAN variants and motivates maximal-margin hyperplane design for adversarial training.
Takeaways & Limitations
The convergence theorem gives only a necessary condition for true and generated samples to become non-separable; sufficiency is left beyond the paper’s scope.
Abstract
from arXiv · showhide
Generative Adversarial Nets (GANs) represent an important milestone for effective generative models, which has inspired numerous variants seemingly different from each other. One of the main contributions of this paper is to reveal a unified geometric structure in GAN and its variants. Specifically, we show that the adversarial generative model training can be decomposed into three geometric steps: separating hyperplane search, discriminator parameter update away from the separating hyperplane, and the generator update along the normal vector direction of the separating hyperplane. This geometric intuition reveals the limitations of the existing approaches and leads us to propose a new formulation called geometric GAN using SVM separating hyperplane that maximizes the margin. Our theoretical analysis shows that the geometric GAN converges to a Nash equilibrium between the discriminator and generator. In addition, extensive numerical results show that the superior performance of geometric GAN.
1 Introduction
GAN training is framed as a minimax game, but accurate discriminators can cause vanishing or unstable generator gradients. Geometric GAN unifies existing variants through three geometric operations and uses a maximal-margin SVM hyperplane.
- Motivation: GAN trains a generator against a discriminator in a minimax game distinguishing real from generated samples.The generator maps random vectors into data space, while the discriminator separates real and generated samples.
- Motivation: Accurate discriminators can make generator gradients vanish, while the non-saturating update can become unstable near denominator singularities.These difficulties motivated alternative objectives such as Wasserstein GAN and broader integral probability metric formulations.
- Geometric formulation: Geometric GAN decomposes adversarial training into separating-hyperplane search, discriminator movement away from it, and generator movement toward it.The generator update follows the separating hyperplane’s normal direction.
- Geometric formulation: Existing GAN variants mainly differ in their separating-hyperplane construction and geometric scaling factors.The paper provides geometric interpretations of GAN, f-GAN, EB-GAN, and W-GAN and discusses their limitations.
- Geometric GAN: Geometric GAN uses an SVM separating hyperplane with maximal margin and outperforms existing GANs in all tested datasets.The reported numerical experiments support the proposed formulation’s superior performance.
2 Related Approaches
The paper interprets mean feature matching GAN training through feature-space distances, constrained updates, and explicit separating-hyperplane geometry. This interpretation motivates comparing classifiers by their normal vectors and selecting a maximal-margin hyperplane.
- 2.1 Mean feature matching GAN: The integral probability metric measures differences between probability distributions over a chosen bounded function space.With finite-Lipschitz functions, it becomes the Wasserstein or earth mover’s distance.
- 2.1 Mean feature matching GAN: McGAN maps random inputs through a generator and defines its function space using a bounded feature map into a higher-dimensional feature space.The resulting function space is symmetric because negating any admissible function remains admissible.
- 2.1 Mean feature matching GAN: McGAN training uses mini-batch empirical optimization with discriminator and generator updates, plus projection and clipping to satisfy constraints.The discriminator update uses stochastic gradient descent, while the generator update follows the updated discriminator parameters.
- 2.2 Geometric interpretation: The explicit McGAN updates reveal a geometry in which the discriminator moves samples away from a separating hyperplane and the generator moves fake samples toward it.The generator follows the hyperplane’s normal-vector direction.
- 2.2 Geometric interpretation: Because linear classifiers are determined by a normal vector and offset, existing GAN variants can be compared through their normal-vector directions and scaling factors.This observation motivates an optimal separating hyperplane with maximal margin.
3 Geometric GAN
Geometric GAN formulates adversarial training around an SVM separating hyperplane that maximizes the margin, with alternating discriminator and generator updates. Its convergence theorem states that an optimal solution matches the data distribution and has cost 2.
- SVM hyperplane: Geometric GAN uses a soft-margin SVM linear classifier to maximize the margin between true and fake samples.The formulation includes a tuning parameter C and slack variables allowing points on the wrong side of the margin.
- SVM hyperplane: The SVM normal vector is determined by support vectors, including points on the margin boundary and points on the wrong side.The support-vector region is defined as M = {φ ∈ Ξ | |⟨w_SVM, φ⟩ + b| ≤ 1}.
- Geometric updates: Discriminator updates push supporting vectors toward the margin boundaries, while generator updates move fake feature vectors toward the separating hyperplane along its normal direction.These updates are expressed through alternating stochastic-gradient steps and geometric scaling factors.
- Geometric updates: The adversarial training alternates discriminator and generator minimization using hinge-style costs based on the linear discriminator.The discriminator cost uses expectations of [1 − D(x)]_+ and [1 + D(g_θ(z))]_+.
- Convergence: At an optimal alternating-minimization solution, the generated distribution equals the data distribution almost everywhere and R(D*, G*) = 2.The theorem gives this as the key convergence result for geometric GAN.
- Convergence: A parallel-lines example reaches its generator-cost minimum at θ* = 0, where true and fake samples lie on the separating hyperplane.The example interprets the minimum cost value 2 as the non-separability condition at equilibrium.
4 Experimental Results
The experiments compare geometric GAN with representative GAN variants on synthetic and image datasets, including settings with Lipschitz constraints. Geometric GAN showed less mode collapse and stable, realistic generation.
- Mixture of Gaussians: Comparisons covered GAN, Wasserstein GAN, mean-difference methods, and geometric GAN on a mixture of 25 Gaussians.The synthetic distribution used 25 Gaussian modes arranged on a 5×5 grid.
- Mixture of Gaussians: Geometric GAN demonstrated the least mode collapsing behavior independently of Lipschitz continuity regularization constraints.Results were illustrated in Figures 4 and 5.
- Mixture of Gaussians: Under identical Lipschitz density constraints, linear hyperplane approaches showed less mode collapse than original GAN through more consistent gradients.The comparison attributes this difference to linear versus nonlinear separating hyperplanes.
- Mixture of Gaussians: Mean-difference hyperplanes in Wasserstein GAN or McGAN led generators toward the mean of an arbitrary number of true-distribution modes.Geometric GAN generally showed robust and consistent convergence toward the true distributions.
- Image Datasets: On MNIST, CelebA, and LSUN, geometric GAN generated realistic images without mode collapsing or divergent behavior.The large-scale comparisons were qualitative because consistent quantitative measures remained under debate.
5 Conclusion
The paper proposes geometric GAN from a geometric interpretation of adversarial training and reports theoretical convergence and improved empirical behavior. Its SVM separating hyperplane maximizes the margin between classes.
- 5 Conclusion: Geometric GAN uses an SVM separating hyperplane with maximal margin between the two classes.The formulation is derived from geometric intuition rather than primarily statistical design criteria.
- 5 Conclusion: Theoretical results show that the proposed algorithm converges to a Nash equilibrium between discriminator and generator.The equilibrium also has a geometric meaning.
- 5 Conclusion: Extensive numerical experiments showed less mode collapsing and more stable training behavior for the proposed method.
A Geometric interpretation of GAN and its variants
The appendix represents the discriminator through a linear output layer over learned features and uses this form to interpret GAN variants geometrically. Their differences arise from separating hyperplanes and geometric scaling factors.
- A Geometric interpretation of GAN and its variants: The discriminator is written as a specific form whose feature-space output uses a linear layer w over Φζ(x), followed by an output activation.The appendix denotes the linear feature score by Vw,ζ(x).
- A Geometric interpretation of GAN and its variants: Existing GAN approaches mainly differ in their separating-hyperplane definitions and geometric scaling factors under this discriminator form.
A.1 GAN
The appendix explains vanilla GAN training through geometric scaling factors applied to true and synthetic feature vectors. It also identifies vanishing generator updates when the discriminator becomes accurate.
- A.1 GAN: Geometric scaling factors are defined for true and synthetic feature vectors in the GAN cost formulation.
- A.1 GAN: With sigmoid activation, the discriminator output uses S(u) = 1/(1 + e^-u), linking the scaling factors to the activation.
- A.1 GAN: The discriminator update separates samples through a separating-hyperplane update, while chain-rule gradients provide the corresponding parameter updates.
- A.1 GAN: Scaling parameters reflect the geometry of curved feature spaces induced by logarithms and nonlinear activations.
- A.1 GAN: As the discriminator becomes accurate and s_i ≃ 0, the generator update becomes more difficult.
A.2 f-GAN
The f-GAN formulation uses divergence-specific geometric scaling factors for true and fake feature vectors within its adversarial updates. These factors differ across f-divergences and are often asymmetric, making update balance important.
- f-GAN defines geometric scaling factors for true and fake feature vectors within its minmax formulation.
- Table 1 summarizes recommended final-layer activations and geometric scaling factors for f-GAN.
- The separating hyperplane update and the discriminator and generator updates are derived from these geometric constructions.
- Except for the total variation-based divergence, f-GAN scaling factors are asymmetric, so controlling discriminator-generator update balance is technically important.
A.3 Wasserstein GAN
Wasserstein GAN minimizes an integral probability metric under a Lipschitz constraint. Its update is largely equivalent to mean matching with unit geometric scaling factors, apart from the hyperplane-normal update.
- Wasserstein GAN minimizes an integral probability metric over functions constrained by a Lipschitz seminorm.
- The W-GAN update differs from mean matching primarily in using a mean difference on an l∞ ball for the hyperplane normal vector update.
- For W-GAN, the geometric scaling factors are t_i = s_i = 1 for all i.
A.4 Energy-based GAN
Energy-based GAN introduces a positive margin into alternating discriminator and generator optimization. Its geometric interpretation links margin-dependent scaling and sample exclusion to the separating-hyperplane updates.
- EBGAN alternates minimization of discriminator and generator cost functions defined using a positive margin m.
- The hinge-like function ψ(y) has subgradient values that depend on whether y lies below, within, or above the margin.
- Margin-dependent geometric scaling factors are defined for true and fake feature vectors before the separating-hyperplane update.
- Unlike geometric GAN, EBGAN excludes only fake samples outside the margins from hyperplane and discriminator updates.
- Geometric GAN instead excludes both true and fake samples outside the margins, a symmetric exclusion observed to improve robustness to mode collapse.
- Compared with regression, geometric GAN obtains its hyperplane from samples within each mini-batch rather than designing a classifier for each sample pair.
B Proof for Theorem 3.1
The proof establishes the optimality conditions for alternating minimization using lemmas about the margin-based objective. It concludes that the generator distribution equals the data distribution almost everywhere.
- Lemma B.1 shows that ϕ(y) = (m − y)+[m + y]+ has minimum 2m for every y ≥ −m.
- Lemma B.2 locates the minimum of α[m − y]+ + β[m + y]+ within [−m, m], at an endpoint determined by α and β.
- The proof applies alternating-minimization conditions to the lower semi-continuous objectives and separates the domain using an indicator function.
- The integral terms attain their minima under density-ordering cases, yielding the lower bound R(D*, g*) ≥ 2.
- Equality holds if and only if p_g*(x) = p_data(x) almost everywhere, completing the proof of the convergence result.