On gradient regularizers for MMD GANs

1 Introduction

• GAN optimization can be difficult because its generator gradient may provide no useful signal when P and Qθ are not absolutely continuous, motivating stabilization methods [2] [38] [42].Source: §1 · p. 1
• Wasserstein IPMs use smooth critics with a uniform Lipschitz constraint that is hard to enforce; a popular approximation constrains gradient norms in expectation [18].IPMs can give credit to Qθ when it is near P [3, 8, Section 4 of 15].Source: §1 · p. 1
• MMD losses use RKHS critics, and learned image features can improve performance; prior MMD GANs have applied weight clipping [3] or gradient penalties [18] to the critic [5] [7] [27].Source: §1 · pp. 1–2
• The regularizer adapts a semi-supervised penalty that combines variance and a density-weighted gradient norm, placing the decision boundary in low-density regions for P and Qθ.The paper instead applies its related penalty to the MMD critic only on P.Source: §1 · p. 2
• The proposed MMD critic penalty applies only on P, keeping the critic flatter where P has high mass without forcing it to vanish on generator samples from Qθ.Excluding Qθ also avoids the minibatch-dependence concern raised by [32]; the resulting discrepancy is asymmetric, not an IPM, and its critic class depends on P.Source: §1 · p. 2

2 Learning implicit generative models with MMD-based losses

• For a fixed bounded kernel with Lipschitz embeddings, MMD is weakly continuous, and a characteristic Gaussian kernel also makes it strict.Continuity means the loss can provide a better signal as Qθ approaches P; the section contrasts this with Jensen–Shannon loss, which can remain constant until it jumps to zero.Source: §2 · pp. 2–3
• MMDs with learned kernel families improve on simple pixel-level kernels, but optimizing the kernel can make the resulting distance discontinuous and create problematic gradient fields.The DiracGAN example shows that the optimized MMD need not converge to zero as the modeled point mass approaches the target; some gradient-field trajectories move in the wrong direction or become stuck.Source: §2 · p. 3, Figure 1 · p. 4
• If K is characteristic and the feature maps φψ distinguish every pair of unequal distributions, the optimized MMD is strictly positive whenever P ≠ Q.The proposition establishes this by choosing a feature map whose pushforward distributions differ, then applying the characteristic-kernel property.Source: §2 · p. 3
• Learned low-dimensional feature maps paired with a simple characteristic kernel can suffice when the feature-map family is rich enough to distinguish unequal distributions.The section describes kψ = K ◦ φψ, with examples of feature dimension s = 16 or s = 1, and says it need not ensure every kψ is characteristic.Source: §2 · p. 3
• Uniformly bounded critics with a common Lipschitz constant make the optimized MMD continuous in the weak topology.The stated sufficient conditions include kernels kψ = K ◦ φψ with bounded K(a, a), Lipschitz kernel embeddings, and uniformly Lipschitz feature maps.Source: §2 · p. 3
• Box constraints on ψ or additive critic-gradient regularization can yield MMD GANs that match or outperform WGAN-based models, but the additive penalty does not substantially change the problematic gradient field.The section uses this limitation to motivate distances with better convergence behavior.Source: §2 · pp. 3–4

3.1 Lipschitz Maximum Mean Discrepancy

• LipMMD restricts critics to Lipschitz functions, is bounded above by Wasserstein distance, and is conjectured to approach it as λ→0 for universal kernels, but is generally difficult to estimate.The optimized LipMMD is also bounded above by Wasserstein distance and is continuous in the Wasserstein topology.Source: §3.1 · p. 4

3.2 Gradient-Constrained Maximum Mean Discrepancy

• GCMMD penalizes critic gradients under μ (experiments use target P) and adds a Sobolev-inspired norm; it yielded good MNIST models but took days even at 28×28.Its estimator uses kernel and derivative matrices, and the paper notes the linear system is impractical even with a low-rank approximation.Source: §3.2 · p. 5
• Unlike Fisher GAN [34] and Sobolev GAN [33], which use additive regularization, GCMMD enforces its constraints in closed form.Sobolev GAN constrains the gradient norm under μ and uses a vanishing boundary condition to define a solution.Source: §3.2 · p. 5

3.3 Scaled Maximum Mean Discrepancy

• SMMD uses a lower bound on GCMMD that can be estimated in time linear in dimension while controlling the critic’s Sobolev and gradient norms.The bound is more restrictive than GCMMD, so SMMD is no greater than GCMMD.Source: §3.3 · pp. 5–6
• In the toy example, optimized SMMD supplies a strong gradient toward the target and is continuous in its location parameter.At a location where the prior loss had an extremely small gradient, SMMD first reduces the critic gradient until a learning signal is available.Source: §3.3 · p. 6
• Under the stated network, kernel, and density conditions, SMMD is Wasserstein-continuous even when μ depends on P and Q; L2(μ) gradient control alone need not bound the critic’s Lipschitz norm.The theorem also assumes bounded weight-matrix condition numbers and other stated smoothness conditions.Source: §3.3 · p. 6
• Spectral parametrization keeps critic weight conditioning stable during SMMD training, whereas conditioning worsens in the default case.The method parameterizes weight matrices using their spectral norm to encourage diverse, higher-rank filters.Source: §3.3 · p. 6

Results

• On CIFAR-10, SN-SWGAN and SN-SMMDGAN performed comparably to SN-GAN, while SN-SWGAN far outperformed WGAN-GP on both CIFAR-10 and CelebA.Source: §4 · p. 7
• On CelebA, SN-SWGAN and SN-SMMDGAN beat same-architecture methods on all three metrics and trained faster; on ImageNet, proposed methods beat BGAN and SN-GAN on FID and KID.The CelebA methods also consistently outperformed other methods over multiple initializations.Source: §4 · pp. 7, 9, Table 1 · p. 9

Spectrally normalized WGANs / MMDGANs

• Spectrally normalized WGAN and Sobolev-GAN failed to converge, while MMDGAN converged on CIFAR-10 but was unstable on CelebA; spectral normalization provided too-loose gradient control for these methods.Figure 2 also shows that SMMD controls the critic's expected gradient better, even with spectral normalization.Source: Figure 2 · p. 8, §4 · p. 9

Rank collapse

• SMMD sometimes suffered critic rank collapse, especially on CelebA, while spectral parametrization appeared to prevent it.The authors also found that increasing RKHS regularization after reverting to an earlier checkpoint could avoid collapse, but did not use that intervention in these experiments.Source: §4 · p. 9

• Gradient-Constrained MMD and its Scaled MMD approximation provide principled gradient control for MMD-based critics and achieve excellent performance in practice.The work also clarifies how previous techniques relate to the DΨ MMD loss.Source: §5 · p. 9
• Future work could study how these distances diffuse particles from Q toward P; Mroueh et al. [33, Appendix A.1] began this for Sobolev GAN, and [35] proved convergence and studied discrete-time approximations.Source: §5 · p. 9
• Future work could examine the geometry of optimized-kernel distances, which may have interesting properties beyond the Wasserstein-versus-MMD comparison studied by Bottou et al. [8].Their results do not address distances based on optimized kernels.Source: §5 · p. 9

A Proofs

• The appendix assumes a separable RKHS with a basis and a twice continuously differentiable kernel, and specifies notation for kernel derivatives.It also lists reproducing properties and Hilbert-Schmidt identities, including the rank-one operator norm and inner product relations.Source: A Proofs · p. 12

A.1 Definitions and estimators of the new distances

• Assumptions (A)–(D) require integrable first moments, at-most-linear growth of k(x, x), twice continuous differentiability, and specified mu-integrability.The integrable functions are x ↦ k(x, x) and x ↦ ∂i∂i+d k(x, x) for 1 ≤ i ≤ d.Source: §A.1 · p. 12
• Under Assumptions (A)–(D), Proposition 5 expresses Gradient-Constrained MMD in terms of the operator D_mu,lambda.The passage states the operator form but does not provide its formula in the supplied text.Source: §A.1 · p. 12
• Proposition 3 gives a computable discrete-measure estimator using the sample kernel matrix K and derivative matrices G and H.G contains left derivatives of the kernel, while H contains derivatives with respect to both arguments.Source: §A.1 · p. 13
• Proposition 4 gives an all-f result under Assumptions (A)–(D), identified as key to defining SMMD.The supplied passage does not include the result’s formula.Source: §A.1 · p. 14
• Lemma 6 establishes that D_mu is trace-class, symmetric, and positive semidefinite, and that D_mu,lambda is invertible for positive lambda.It also gives an inner-product identity for Hilbert-Schmidt operators.Source: §A.1 · p. 15
• For distributions with finite first moment, k(x, ·) is Bochner integrable and its expected feature-map inner product equals the inner product with its mean embedding.The passage applies this to P and Q to express their expected difference through the difference in mean embeddings.Source: §A.1 · pp. 13, 15

A.2 Continuity of the Optimized Scaled MMD in the Wasserstein topology

• The theorem assumes bounded per-layer condition numbers, an absolutely continuous input distribution, decreasing layer widths, leaky ReLU activations, and a top-level kernel satisfying Lipschitz and additional regularity conditions.Spectral parametrization can enforce the condition-number constraint; orthogonal normalization [11] is another option.Source: §A.2 · p. 16
• The assumptions ensure differentiability almost everywhere and a locally full-rank network; the kernel conditions hold for smooth translation-invariant kernels such as Gaussian, and for the linear kernel.Absolutely continuous μ avoids nondifferentiable points with positive probability; decreasing widths and leaky ReLU help ensure local full rank almost everywhere.Source: §A.2 · p. 16
• Under Assumptions (I)–(V), Theorem 1 establishes continuity of the optimized scaled MMD in the Wasserstein topology.The proof concludes that scaled MMD follows from the Wasserstein bounds developed in the section.Source: §A.2 · pp. 16–17
• MMD_k(P, Q) ≤ W_dk(P, Q), and the proof bounds W_dψ(P, Q) by L_K∥φ_ψ∥_Lip W(P, Q).Here W is the standard Wasserstein distance.Source: §A.2 · p. 17

A.2.1 When some of the assumptions don’t hold

• As α → 0, E_μ[∥∇φ_α(X)∥²] → 0, but the squared Lipschitz constant converges to 2(1 − γ)².In Figure 4, the areas of regions A3 and A4 become negligible as α approaches zero.Source: §A.2.1 · p. 20, Figure 4 · p. 20
• As β → 0, E_μ[∥∇φ(X)∥²] → 0 while ∥φ∥²_L converges to (1 − γ)², despite W_β having a bounded condition number.At β = 0, ∇φ_0(X) = 0 for all X ∈ R², even though the matrices remain full rank.Source: §A.2.1 · p. 21

B DiracGAN vector fields for more losses

• Figure 5 compares parameter vector fields for several losses in Example 1, with trajectories beginning from three initial parameter values.The setup follows [30], with P = δ0, Q = δθ, and φψ(x) = ψx.Source: Figure 5 · p. 21, B DiracGAN vector fields for more losses · p. 21
• The plotted alternatives include MMD-GP-unif, MMD-GP, SN-MMD, Sobolev-MMD, and GCMMDμ,k,λ.MMD-GP-unif uses a gradient penalty sampled between P and Q samples, MMD-GP takes the expectation under P, SN-MMD fixes ψ = 1, Sobolev-MMD uses the quadratic penalty, and GCMMDμ,k,λ uses μ = N(0, 10²).Source: Figure 5 · p. 21, B DiracGAN vector fields for more losses · p. 22

C Vector fields of Gradient-Constrained MMD and Sobolev GAN critics

• Sobolev critic gradient norms become orders of magnitude higher outside the distributions’ high-density regions, unlike Gradient-Constrained MMD gradients.Figure 6 compares both vector fields for two Gaussians, using μ = (P + Q)/2 for both.Source: Figure 6 · p. 22, C Vector fields of… · p. 22
• The Sobolev critic’s very high boundary gradients arise from its vanishing boundary condition and point opposite to the target distribution.The paper attributes this behavior to the boundary condition imposed by Sobolev GAN.Source: C Vector fields of… · p. 22

D An estimator for Lipschitz MMD

• The Lipschitz MMD estimator enforces H_k ≤ 1 at m densely spaced points around the supports of P and Q, rather than everywhere in X.It estimates the discrepancy from samples X ∼ P_nX and Y ∼ Q_nY.Source: D An estimator for Lipschitz MMD · pp. 22–23
• The generalized representer theorem gives the optimal f a finite form, making the objective linear in its coefficients and the constraints quadratic.The quadratic constraints are constructed from kernel evaluations on concatenated X and Y samples and kernel derivatives at the constraint points.Source: D An estimator for Lipschitz MMD · p. 23
• The resulting linear optimization with convex quadratic constraints is solvable by standard software, but requires dense coverage and substantial computation even for a one-dimensional toy problem.The approximation is reasonable only when the points effectively cover the region of interest.Source: D An estimator for Lipschitz MMD · p. 23
• Derivatives with respect to input points or kernel parameters are almost free after estimation if the solver supplies dual variables for the constraints.The method uses the envelope theorem and differentiates the KKT conditions, as in [1].Source: D An estimator for Lipschitz MMD · p. 23

E Near-equivalence of WGAN and linear-kernel MMD GANs

• When E_P φ(X) > E_Q φ(Y), linear-kernel MMD equals the WGAN loss, and its gradient penalty acts on the same critic function.This equivalence is stated under the expectation-ordering assumption used as the goal of critic training.Source: E Near-equivalence of WGAN and… · pp. 23–24
• The equivalence is not exact for typical MMD GAN training with an unbiased MMD^2 estimator; [7] also penalized η rather than the true critic η/∥η∥.Source: E Near-equivalence of WGAN and… · p. 24
• Linear-kernel SMMD is analogous to applying a scaling operator to a WGAN, motivating the name SWGAN.Source: E Near-equivalence of WGAN and… · p. 24

F.1 Comparison of Gradient-Constrained MMD to Scaled MMD

• For Gaussian distributions, MMD ∝ SMMD and Gradient-Constrained MMD behaved similarly, and optimizing the latter two was similar, while optimizing MMD yielded an essentially constant distance.Source: §F.1 · p. 24

F.2 IGMs with Optimized Gradient-Constrained MMD loss

• The Proposition 3 estimator used an empirical mean for η, shared samples for μ = P, and incomplete Cholesky decomposition [43, Algorithm 5.12] for its approximately low-rank matrix system.The decomposition produced R such that the system was approximated by RᵀR, enabling use of the Woodbury matrix identity.Source: §F.2 · p. 24
• Despite a good approximation with small ℓ and no need to construct full matrices, slow, GPU-unfriendly backpropagation limited the estimator to MNIST and made optimization take days.Training on CPU was faster than on GPU for this procedure.Source: §F.2 · p. 24
• MNIST models were reasonable, but training later showed rank collapse; spectral normalization may have delayed, not prevented, it, and SMMD converged somewhat faster with the estimator comparison still uncertain.The authors note that the convergence difference may reflect the Proposition 3 estimator rather than the distances, and that further comparison is needed.Source: §F.2 · p. 24

F.3 Spectral normalization and Scaled MMD

• MMDGAN without spectral normalization or a gradient penalty did surprisingly well in this case, though it fails badly in other situations.Source: §F.3 · p. 24
• On CelebA using SMMD, Figure 9 compares critic-layer singular-value decay early and later in training with and without spectral parametrization.Figure 9 shows singular-value distributions across more critic layers.Source: §F.3 · p. 24, Figure 9 · p. 26
• Spectral parametrization alone is often insufficient for good results in comparisons of FID and KID across Sobolev-GAN, MMDGAN, and spectral-normalization variants.The cited passage identifies the metrics and model families but supplies no specific scores.Source: §F.3 · p. 26

• Figures 12 and 13 provide extra samples from the models.Source: §F.4 · p. 26
• Figure 12 shows samples from an ImageNet generator trained with Scaled MMD and Spectral Normalization (SN-SMMDGAN).Source: Figure 12 · p. 28
• Figure 13 compares samples from different models trained on CelebA at 160×160 resolution.Source: Figure 13 · p. 29