PaperlayerView exact revision on arXiv

Source-linked AI summary

McGan: Mean and Covariance Feature Matching GAN

Youssef Mroueh, Tom Sercu, Vaibhava Goel

arXiv:1702.08398v2cs.LGstat.ML

TL;DR

GAN training requires an objective for comparing generated and real distributions, while standard generative-model evaluation is difficult. The paper introduces IPMs that match mean and covariance statistics in a finite-dimensional feature space, and reports stable training, reduced mode dropping, and an IPM loss correlated with generated-sample quality. The authors limit their experimental claims mainly to training stability and comparable sample quality, while noting unresolved alternatives to feature-map clipping.

  • Problem

    GAN training needs meaningful objectives for comparing the learned generator distribution with the real data distribution, and generative-model evaluation is difficult.

  • Method

    The paper embeds distributions in a finite-dimensional feature space and trains GANs by matching mean and covariance feature statistics through IPMs.

  • Results

    McGan variants are stable to train, show reduced mode dropping, and have IPM losses that correlate with generated-sample quality.

  • Takeaways & Limitations

    Mean and covariance feature matching provide IPM-based objectives for GAN training, with mean matching and covariance matching each admitting formulations usable as training objectives.

  • Takeaways & Limitations

    The experiments focus mainly on training stability and avoid claiming better sample quality, while alternatives to clipping that ensure bounded feature maps remain an open question.

Abstract

from arXiv · show

We introduce new families of Integral Probability Metrics (IPM) for training Generative Adversarial Networks (GAN). Our IPMs are based on matching statistics of distributions embedded in a finite dimensional feature space. Mean and covariance feature matching IPMs allow for stable training of GANs, which we will call McGan. McGan minimizes a meaningful loss between distributions.

1. Introduction

The paper develops mean and covariance feature-matching IPMs for GANs by embedding distributions in finite-dimensional feature spaces. McGan is presented as stable to train, with reduced mode dropping and an IPM loss correlated with generated-sample quality.

  • 1. Introduction: The paper proposes matching distribution means and covariances after embedding distributions in a finite-dimensional feature space.The construction is inspired by MMD and kernel mean embeddings.
  • 1. Introduction: Mean feature matching IPMµ,q has equivalent primal and dual formulations and can serve as a GAN training objective.The formulation supports any ℓq norm for matching means.
  • 1. Introduction: Wasserstein GAN’s parametrization corresponds to ℓ1 mean feature matching GAN, denoted IPMµ,1 GAN in this framework.
  • 1. Introduction: Covariance feature matching IPMΣ has two dual formulations and can also be used as a GAN training objective.The covariance formulation uses a Ky-Fan norm that can be computed without explicitly constructing full covariance matrices.
  • 1. Introduction: McGan variants are stable to train, show reduced mode dropping, and have IPM losses that correlate with generated-sample quality.

2. Integral Probability Metrics

Integral Probability Metrics define distances between distributions through a class of bounded measurable functions. The paper uses IPMs to formulate generative modeling as minimizing the discrepancy between generated and real distributions, including an empirical min-max objective.

  • 2. Integral Probability Metrics: An IPM measures the distance between two probability distributions using a chosen function space of bounded measurable real-valued functions.The function space determines which distributional differences the IPM captures.
  • 2. Integral Probability Metrics: For symmetric function spaces, the IPM can be written without the absolute value because every function has its negative in the space.
  • 2. Integral Probability Metrics: IPMs are pseudo-metrics: they satisfy non-negativity, symmetry, and the triangle inequality, but zero distance need not imply identical distributions.
  • 2. Integral Probability Metrics: Choosing different function spaces yields different probability-measure distances and can make generative-model learning tractable.
  • 2. Integral Probability Metrics: The generative model maps samples from a fixed latent distribution through gθ to produce a distribution Pθ close to the real distribution Pr.Training is formulated using an IPM between Pθ and Pr.
  • 2. Integral Probability Metrics: Given real and latent samples, the paper defines an empirical optimization problem for training the generative model.For simplicity, the discussion sets the number of real and latent samples equal, M = N.

3. Mean Feature Matching GAN

Mean feature matching embeds distributions in a finite-dimensional learned feature space and compares their mean embeddings. The resulting IPM has primal and dual GAN formulations, supports adversarial training, and provides a framework encompassing several prior GAN objectives.

  • Mean feature representation: The function class uses bounded linear functions ⟨v, Φω(x)⟩ over a nonlinear finite-dimensional feature map.The feature map is typically parameterized by a neural network, with constraints chosen to keep the function space bounded.
  • Mean matching IPM: The induced IPM equals the worst-case ℓq distance between the distributions’ mean feature embeddings.The maximum is taken over the feature-map parameters ω.
  • GAN objective: Mean matching GAN training admits primal and dual formulations with an adversarial interpretation: the feature space separates mean embeddings while the generator brings them together.The primal and dual formulations are introduced as equivalent ways to optimize the mean matching objective.
  • Computational considerations: The primal formulation is more computationally appealing because its linear expectation structure supports stochastic gradients, whereas the dual requires a large real-data minibatch for expectation estimates.The dual formulation needs real and fake samples for training, while the primal does not need real samples when optimizing the generator.
  • Connections to prior GANs: The framework shows that WGAN, MMD GAN, and Improved GAN objectives can be expressed within or closely related to mean feature matching.WGAN’s particular parametrization corresponds to a mean feature matching IPM, while finite-dimensional learned-feature IPMµ,2 is a special case of MMD.

4. Covariance Feature Matching GAN

Covariance feature matching extends mean matching by comparing second-order statistics in a finite-dimensional feature space. Its IPM selects discriminative covariance directions, yielding a generator–feature-map adversarial game that motivates McGan.

  • Covariance representation: Covariance feature matching incorporates second-order information by comparing covariance embeddings of distributions in feature space.The construction is motivated by representing a distribution approximately through its mean and covariance.
  • Covariance matching IPM: The covariance IPM finds k directions that maximize discrimination between the two covariance embeddings.These directions play a role analogous to principal directions, but maximize discrimination between covariance matrices rather than variance within one distribution.
  • Covariance matching IPM: The resulting distance is the Ky Fan k-norm, or truncated nuclear norm, of the covariance-feature difference.The matrices U and V are the left and right singular vectors of the covariance difference.
  • GAN objective: Covariance matching GAN training is an adversarial game in which the feature map maximizes covariance discrepancy and the generator minimizes it.This dual formulation motivates the name covariance matching GAN.
  • Mean and covariance matching: Mean and covariance statistics are combined so the discriminator distinguishes real and fake means and variances while the generator matches the real statistics.The paper presents this extension as the mean and covariance feature matching GAN, or McGan.

5. Algorithms

The paper provides algorithms for mean and covariance feature matching GANs. The covariance primal algorithm alternates stochastic ascent for the feature-side variables with descent for the generator while enforcing boundedness and orthonormality constraints.

  • Algorithm overview: The algorithms cover both mean feature matching IPMµ,q and covariance feature matching IPMΣ GANs.These are presented as the algorithms for mean and covariance feature matching GAN, or McGan.
  • Mean Matching GAN: Mean Matching GAN uses critic updates with projection onto an ℓp ball and clipping of ω to keep the feature map bounded.The algorithm alternates critic optimization with generator updates using minibatches of real and latent samples.
  • Mean Matching GAN: The mean dual algorithm requires real and fake samples to train both the generator and the critic feature space.The algorithm is described as using samples from both distributions during training.
  • Covariance Matching GAN: Covariance Matching GAN performs stochastic gradient ascent on (ω, U, V) and descent on θ.It clips ω to ensure boundedness of Φω and applies QR retraction on the Stiefel manifold to preserve orthonormality of U and V.

6. Experiments

Experiments evaluate mean and covariance matching GANs on image-generation datasets, emphasizing training stability, loss behavior, and sample quality. The study also examines primal and dual formulations, conditional generation, and practical training considerations.

  • Evaluation: The experiments use random, non-cherry-picked samples and avoid claiming better sample quality because generative-model evaluation is difficult.The authors focus primarily on training stability and use samples to claim comparable rather than superior quality.
  • Formulations: Mean-matching GANs were evaluated in primal and dual forms, while covariance matching used the primal formulation because the dual was less efficient and infeasible for covariance matching.The primal IPMµ,1 formulation corresponds to the original WGAN parameterization.
  • Mean matching: McGan training was stable, with steadily decreasing IPM losses across primal and dual mean-matching formulations and ℓq norms.Samples from shared noise remained similar in identity or style while details evolved during training.
  • Optimization: Dual mean matching required a threefold larger real-data minibatch to obtain a good estimate for the generator gradient.The real minibatch was increased to 3 × 64.
  • Covariance matching: Covariance matching produced samples of comparable quality to mean matching and remained stable as the number of components varied from 4 to 64.Experiments used k = 16 components for the reported lfw and LSUN covariance-matching results.
  • Conditional generation: Conditional CIFAR-10 experiments combined IPM and cross-entropy objectives, producing recognizable class-conditioned samples while shared noise influenced visual style.Rows correspond to the same class and columns share the same random noise.

7. Discussion

The discussion identifies critic clipping as a capacity constraint and points to alternatives and extensions for future work.

  • Discussion: Clipping reduced critic capacity, requiring more feature maps to compensate.The authors also report successful use of an ℓ2 weight penalty to maintain boundedness.

Supplementary Material for McGan: Mean and Covariance Feature Matching GAN

The supplied supplementary-material passage identifies the paper’s authors.

  • Authors: The listed authors are Youssef Mroueh, Tom Sercu, and Vaibhava Goel.The passage marks Mroueh and Sercu with asterisks and lists affiliation indices 1 and 2.

A. Subspace Matching Interpretation of Covariance Matching GAN

The covariance discrepancy Δω is analyzed through its eigenvalues and singular values, revealing that the simplified energy form δEω is not itself an IPM. Nevertheless, δE can be optimized as a lower bound of IPMΣ with an energy-based interpretation.

  • Covariance discrepancy: Δω is symmetric but not positive semidefinite, and its singular values satisfy σj = |λj| with eigenvector relations uj = sign(λj)vj.These properties motivate simplifying the covariance matching formulation by avoiding separate U and V factors at the optimum.
  • Validity of δEω: The proposed δEω is not guaranteed to be nonnegative because its eigenvalue sum can be negative, so it does not define an IPM.The lack of guaranteed nonnegativity is the decisive distinction from a valid integral probability metric.
  • Optimization interpretation: Although δE is not an IPM, it can be optimized as a lower bound of IPMΣ.This preserves a usable optimization objective while weakening the formal interpretation of the quantity being optimized.
  • Energy interpretation: In its energy interpretation, the discriminator selects a subspace with higher energy on real data than fake data, while the generator maximizes its energy in that subspace.The subspace interpretation connects the covariance-based objective to an energy-based GAN formulation.

B. Mean and Covariance Matching Loss Combinations

The paper compares mean-matching, covariance-matching, and combined IPM objectives through class-conditioned Cifar-10 samples. Across these configurations, the same noise is used within columns and the same class conditioning within rows, while the reported architecture produced qualitatively good samples with IPMΣ but WGAN failed in one matched configuration.

  • Experimental setup: All McGan sample results use the same generator and discriminator architecture, which produced qualitatively good samples with IPMΣ.The reported samples combine different IPMµ,q and IPMΣ objectives under this common architecture choice.
  • Reported comparison: With the same hyperparameters and architecture, WGAN failed to produce good samples in Figure 7, although training converged in other configurations.The passage reports this as a configuration-specific observation rather than a general failure of WGAN training.
  • Mean matching: Figure 7 compares class-conditioned samples generated with IPMµ,1(WGAN), using shared noise within columns and shared class conditioning within rows.Rows progress from airplane through truck, while each column preserves the same random noise z across classes.
  • Mean matching: Figure 8 shows class-conditioned samples generated with IPMµ,2 under the same row and column organization.The rows represent the ten Cifar-10 classes and the columns share random noise z.
  • Covariance matching: Figure 9 shows class-conditioned samples generated with IPMΣ using shared noise across columns and shared class conditioning across rows.This is the covariance-matching configuration associated with the architecture’s qualitatively good samples.
  • Combined objectives: Figures 10 and 11 show class-conditioned samples for IPMµ,1 + IPMΣ and IPMµ,2 + IPMΣ, respectively, with the same row and column controls.These figures represent combined mean-and-covariance matching objectives across the same ten Cifar-10 classes.
Loading 1702.08398v2…