A Characteristic Function Approach to Deep Implicit Generative Modeling

Abdul Fatir Ansari; Jonathan Scarlett; Harold Soh

Source-linked AI summary

A Characteristic Function Approach to Deep Implicit Generative Modeling

Abdul Fatir Ansari, Jonathan Scarlett, Harold Soh

arXiv:1909.07425v2Sep 16, 2019cs.LGcs.NEstat.ML

TL;DR

The paper addresses unstable and computationally costly discrepancy measures for implicit generative modeling. It trains GANs by minimizing a weighted characteristic-function distance, with an optimized weighting distribution variant. The resulting approach is reported to outperform WGAN and MMD-GAN variants while remaining simple and stable to train.

Problem
GAN training can suffer from optimization instability and mode collapse, while MMD-based discrepancies require quadratic time in the number of samples.
Method
The paper trains implicit generative models by minimizing a weighted distance between real and generated characteristic functions, optionally optimizing the weighting distribution parameters.
Results
Experiments report that CFD-based models outperform WGAN and MMD-GAN variants on four benchmark image datasets and other evaluations.
Takeaways & Limitations
Characteristic functions provide an effective alternative for training implicit generative models, with a loss that is simple to implement and stable to train.
Takeaways & Limitations
The empirical CFD may produce high-variance gradient estimates with few sampled frequencies, and optimizing the weighting family requires care to avoid fitting sampling fluctuations.

Abstract

from arXiv · show

Implicit Generative Models (IGMs) such as GANs have emerged as effective data-driven models for generating samples, particularly images. In this paper, we formulate the problem of learning an IGM as minimizing the expected distance between characteristic functions. Specifically, we minimize the distance between characteristic functions of the real and generated data distributions under a suitably-chosen weighting distribution. This distance metric, which we term as the characteristic function distance (CFD), can be (approximately) computed with linear time-complexity in the number of samples, in contrast with the quadratic-time Maximum Mean Discrepancy (MMD). By replacing the discrepancy measure in the critic of a GAN with the CFD, we obtain a model that is simple to implement and stable to train. The proposed metric enjoys desirable theoretical properties including continuity and differentiability with respect to generator parameters, and continuity in the weak topology. We further propose a variation of the CFD in which the weighting distribution parameters are also optimized during training; this obviates the need for manual tuning, and leads to an improvement in test power relative to CFD. We demonstrate experimentally that our proposed method outperforms WGAN and MMD-GAN variants on a variety of unsupervised image generation benchmarks.

1. Introduction

The paper introduces CFD as a characteristic-function-based loss for training implicit generative models, addressing GAN training difficulties with a simple, efficient metric. It establishes theoretical support and reports favorable results against WGAN and MMD-GAN variants.

The work targets GAN optimization instability and mode collapse, which motivate alternative training metrics and schemes.Prior work links uninformative gradients near distributional agreement to the Jensen-Shannon divergence used by the original GAN [14].
CFD trains implicit generative models by minimizing the expected discrepancy between real and generated characteristic functions.The loss is approximately computable in linear time in the number of samples, unlike quadratic-time MMD.
CFD has continuity and almost-everywhere differentiability with respect to generator parameters, plus continuity in the weak topology.
The method is evaluated systematically on synthetic distributions and four image datasets: MNIST, CIFAR10, STL10, and CelebA.
Experiments report that CFD-based GANs outperform WGAN and MMD-GAN variants on quantitative evaluation metrics.The authors also characterize the approach as simple to implement and stable to train.

2. Probability Distances and GANs

GANs use a generator and critic to make generated samples indistinguishable from real data. Modern formulations cast training as minimizing a distributional divergence, including Wasserstein, Cramer, MMD, and Sobolev IPM distances.

A GAN generator transforms latent vectors into data-space samples, while the critic distinguishes generated samples from true data.
Recent GAN formulations express the adversarial game as minimizing a divergence between real and generated distributions.
The original GAN uses Jensen-Shannon divergence, while later methods use Wasserstein, Cramer, MMD, and Sobolev IPM distances.The cited alternatives include the 1-Wasserstein distance [16], Cramer distance [4], MMD [24] [5] [1], and Sobolev IPM [31].

3. Characteristic Function Distance

The characteristic function distance integrates squared discrepancies between characteristic functions under a weighting distribution. Its empirical form uses sampled frequencies, and optimizing the weighting distribution can improve test power when it targets informative frequencies.

3. Characteristic Function Distance: CFD is the weighted integrated squared error between the characteristic functions of distributions P and Q.Characteristic functions uniquely determine distributions, and the weighting function must make the integral converge.
3. Characteristic Function Distance: When the weighting function is a probability density, CFD becomes an expectation over sampled frequencies; if its support is all of R^d, zero CFD is equivalent to P = Q.
3. Characteristic Function Distance: ECFD approximates CFD with empirical characteristic functions computed from data samples and frequencies drawn from the weighting distribution.The empirical characteristic function is formed from equal-weighted characteristic functions of the observed samples.
3.1. Optimized ECFD for Two-Sample Testing: The weighting family and optimized parameters require care because excessive optimization can fit sampling fluctuations and reduce correct null acceptance.Directly optimizing frequencies severely degraded the ability to correctly accept P = Q in the reported example.
3.1. Optimized ECFD for Two-Sample Testing: Optimizing weighting-distribution parameters improves ECFD test power, especially at higher dimensionalities, without impairing correct acceptance of the null in the reported experiment.The experiment uses high-dimensional Gaussians whose means differ in one dimension and compares optimized and smoothed variants.
3.1. Optimized ECFD for Two-Sample Testing: The optimized weighting distribution increases variance more strongly in dimensions where the distributions differ, compensating for frequency-domain discrepancies away from the origin.

4. Implicit Generative Modeling using CFD

CF-GAN trains implicit generative models by maximizing a critic-based characteristic function distance, with optimized weighting-distribution parameters optionally included. The approach has continuity and differentiability guarantees under Lipschitz and integrability assumptions, relates formally to MMD, and avoids MMD’s quadratic computation while retaining strong empirical performance.

Model formulation: CF-GAN uses a critic to maximize CFD between real and generated distributions in a learned lower-dimensional space while the generator minimizes the empirical CFD.The weighting-distribution parameters η may be optimized jointly, or held fixed; in experiments η includes a scale parameter σ.
Theoretical properties: CFD is continuous and differentiable almost everywhere in generator parameters, supporting gradient-based training under the stated assumptions.The continuity theorem assumes local Lipschitz behavior of fφ ◦ gθ and a uniformly finite expected frequency norm.
Theoretical properties: CFD is continuous in the weak topology, so convergence of distributions implies convergence of the metric under the theorem’s Lipschitz and integrability conditions.These results concern general convergent distributions, not only distributions generated by gθ(z).
Assumptions and safeguards: The continuity results require Lipschitz assumptions, and unbounded optimized σ is normalized by ∥σ∥ to prevent it from diverging.Feed-forward generators with ReLU activations satisfy the local Lipschitz condition, while Gaussian, Student-t, and Laplace weighting distributions with fixed σ have finite expected frequency norms.
Relation to MMD: CFD is formally equivalent to squared MMD when the weighting distribution is the inverse Fourier transform of the kernel, but CFD takes O(nk) rather than quadratic sample time.Despite formal equivalence under those conditions, the paper reports different convergence profiles and model performance when optimizing empirical CFD and MMD.
Relation to prior work: Compared with IKL-GAN [25], CF-GAN uses a fixed frequency distribution with learned scale, avoids kernel evaluation, and reports state-of-the-art performance with fewer hyperparameters.The paper presents its direct proofs for optimized CFD properties as distinct from proofs based on CFD–MMD equivalence.

5. Experiments

Experiments compare CF-GAN variants with WGAN and MMD-GAN baselines on synthetic and image datasets, evaluating convergence, FID, KID, and precision-recall. OCF-GAN-GP generally provides the strongest reported performance, while optimized weighting and gradient penalties improve results.

5.1. Synthetic Data: On synthetic distributions, OCF-GAN-GP converges fastest and reaches a better minimum on the complex bimodal dataset D2.For D1, it reaches the same MAE as MMD-GAN-GP, although MMD-GAN-GP converges faster.
5.2. Image Generation: OCF-GAN-GP outperforms WGAN and MMD-GAN variants on FID and KID across CIFAR10, STL10, and CelebA.OCF-GAN with one weighting function outperforms MMD-GAN models using mixtures of five kernels.
5.1. Synthetic Data: Optimizing the scale parameter improves performance, while gradient penalty significantly improves score values across models.The scale optimization also leads to faster convergence in the synthetic experiments.
5.2. Image Generation: Gradient penalty improves recall more than precision across datasets, with the largest recall gains on CIFAR10 and STL10.The authors relate this pattern to greater flexibility than weight clipping, while CelebA shows a smaller difference.
5.2. Image Generation: OCF-GAN-GP generates visually appealing samples across CIFAR10, STL10, MNIST, and 128 × 128 CelebA with a ResNet generator.The qualitative results indicate applicability to larger images and networks.
5.2. Image Generation: Weighting-distribution choices cause no drastic performance changes, but the best choice depends on the dataset and Lipschitz regularization.The overall framework remains robust to reasonable weighting-distribution choices.

6. Discussion and Conclusion

The paper concludes that weighted characteristic-function distances provide an effective alternative for training implicit generative models, with favorable theory and benchmark performance. It also identifies variance and gradient-regularization questions as directions for further work.

CFD-based models outperform MMD-GAN and WGAN variants on four benchmark image datasets.
The proposed weighted characteristic-function distance has attractive theoretical properties for training implicit generative models.
Empirical CFD training may produce high-variance gradient estimates with few sampled frequencies, motivating further study of convergence and gradient regularization.
Optimized CFD loss functions may also serve as alternatives for two-sample-test applications such as domain adaptation and domain separation.

A.1. Proof of Theorem 1

The proof establishes local Lipschitz continuity of the CFD with respect to generator parameters, which implies continuity and almost-everywhere differentiability.

The argument uses bounded characteristic-function differences, local Lipschitzness of the composed generator, and expectation inequalities to obtain the bound.
The proof bounds the CFD difference by 4E_ω*(t)[∥t∥]E_z[L(θ,z)]∥θ−θ′∥ for sufficiently close θ′.
Because the CFD is locally Lipschitz, it is continuous and differentiable almost everywhere by Rademacher’s theorem [13].

A.2. Proof of Theorem 2

The proof shows continuity in the weak topology by reducing the CFD bound to expectations of a bounded Lipschitz function and applying the Portmanteau theorem.

Jensen’s inequality and expectation-order interchange reduce the preceding CFD bound to the convergence of E[g(∥x_n−x∥)].
The proof constructs g(u)=min{2,T_maxL_f|u|}, a bounded Lipschitz function suitable for the Portmanteau theorem.
Convergence in distribution P_n ⇒ P implies uniform convergence of the weighted CFD over the considered parameter class.

A.3. Discussion on an “only if” Counterpart to Theorem 2

The converse implication requires sufficiently rich feature functions and sufficiently well-behaved weighting distributions. Without those conditions, CFD can fail to distinguish distributions.

If the feature class contains only the zero function, every distribution has zero CFD after projection.
If the weighting distribution has bounded support, distributions differing only outside that support may still have zero CFD.
The converse is positive under assumptions including all L-Lipschitz linear functions and a weighting distribution supported on R^m.
Gaussian weighting distributions with fixed positive-definite covariance satisfy the stated support condition, while ReLU networks can implement the required linear functions when their weights are bounded.
The proof uses linear projections and the Cramér-Wold theorem [10] to show that characteristic-function discrepancies remain detectable under the stated assumptions.

B.1. Synthetic Data Experiments

The synthetic experiments construct unimodal and bimodal target distributions, then compare GAN variants using transformation recovery and image-generation metrics. Architectures, weighting distributions, penalties, and evaluation procedures are specified across the experiments.

Synthetic distributions: Synthetic targets include a scale-shift unimodal distribution D1 and a scale-split-shift bimodal distribution D2 generated from Gaussian samples.D1 shifts the mean to −10, while D2 creates modes near −10 and +10.
Synthetic distributions: Because two transformation functions can produce the same target distribution, generator recovery is evaluated with mean absolute error against both valid transformations.The expectations in the MAE equation are estimated using 5000 samples.
Experimental setup: The experiments compare WGAN-GP, MMD-GAN variants, and CF-GAN variants using specified MLP architectures, weighting distributions, gradient penalties, and FID, KID, and precision-recall metrics.FID and KID use Inception features, while MNIST uses LeNet5 features; precision-recall separates sample quality from distribution coverage.
Model variants: Removing the CF-GAN decoder did not affect performance, while reducing the feasible critic set improved training stability, especially for weight-clipped models.The decoder-free OCF-GAN-GP results support the decoder-removal observation.

C. Additional Results

Additional experiments evaluate ECFD variants and generated samples across several image datasets. Optimizing weighting-distribution parameters preserves correct null-hypothesis acceptance, while gradient-penalty models produce sharper samples with fewer artifacts than their non-penalized counterparts.

Two-sample testing: Optimizing weighting-distribution parameters does not reduce the probability of correctly accepting P = Q when the null hypothesis is true.Figure 7 reports this behavior across dimensions and ECFD variants.
Quantitative evaluation: Tables 3 and 4 report FID and KID scores for CIFAR10, STL10, CelebA, and MNIST, with lower scores preferred and averages over five runs.Table 3 includes smoothed ECFD and Laplace and Uniform weighting variants; Table 4 covers MNIST.
Image samples: Gradient-penalty models generate sharper images with fewer artifacts than WGAN and MMD-GAN models without gradient penalties across CIFAR10, CelebA, and STL10.The comparison is shown in the generated-sample figures for the three datasets.
Image samples: OCF-GAN-GP sample quality on MNIST is imperceptibly changed even when the number of random frequencies is reduced to k = 1.Figure 11 varies k for OCF-GAN-GP(N).
Image samples: Additional OCF-GAN-GP samples are provided for 128 × 128 CelebA using a ResNet generator.Figure 12 documents this higher-resolution generator configuration.