Approximation and Convergence Properties of Generative Adversarial Learning

Shuang Liu; Olivier Bousquet; Kamalika Chaudhuri

Source-linked AI summary

Approximation and Convergence Properties of Generative Adversarial Learning

Shuang Liu, Olivier Bousquet, Kamalika Chaudhuri

arXiv:1705.08991v1May 24, 2017cs.LGstat.ML

TL;DR

The paper addresses uncertainty about how restricted discriminators affect GAN approximation and when minimizing an objective guarantees convergence to the target. It defines adversarial divergences to unify objectives and analyzes moment matching and convergence. Under stated conditions, restricted discriminators yield moment-matching effects, while strict adversarial divergences turn objective convergence into weak convergence.

Problem
It is unclear how restricting the discriminator family affects approximation quality or when convergence to an objective’s global minimum implies convergence to the target distribution.
Source: §1 · p. 2
Method
The paper defines adversarial divergences and a strict subclass to study discriminator restriction and distributional convergence across GAN objectives.
Source: §1 · p. 2
Results
Under certain conditions, restricted discriminators yield moment matching, and convergence under a strict adversarial divergence implies weak convergence to the target.
Source: §1 · pp. 2–3
Takeaways & Limitations
The results extend convergence guarantees from isolated objectives to a broad class of GANs and give restricted discriminator families a moment-matching interpretation.
Source: §1 · pp. 2–3
The results do not compare strict adversarial divergences when neither is equivalent to Wasserstein distance or the trivial divergence.
Source: §6 · p. 12

Abstract

from arXiv · show

Generative adversarial networks (GAN) approximate a target data distribution by jointly optimizing an objective function through a "two-player game" between a generator and a discriminator. Despite their empirical success, however, two very basic questions on how well they can approximate the target distribution remain unanswered. First, it is not known how restricting the discriminator family affects the approximation quality. Second, while a number of different objective functions have been proposed, we do not understand when convergence to the global minima of the objective function leads to convergence to the target distribution under various notions of distributional convergence. In this paper, we address these questions in a broad and unified setting by defining a notion of adversarial divergences that includes a number of recently proposed objective functions. We show that if the objective function is an adversarial divergence with some additional conditions, then using a restricted discriminator family has a moment-matching effect. Additionally, we show that for objective functions that are strict adversarial divergences, convergence in the objective function implies weak convergence, thus generalizing previous results.

1 Introduction

The paper addresses two open questions about GAN approximation: how discriminator restrictions affect quality, and when objective convergence implies convergence to the target. It unifies a broad class of GAN objectives and establishes moment-matching and weak-convergence results under stated conditions.

Motivation

The paper frames the questions against limited theoretical understanding of GAN approximation, despite practical success and abundant variants.The questions concern discriminator-family restrictions and whether minimizing different objectives guarantees distributional convergence.Source: §1 · pp. 1–2

Unified framework

The framework defines adversarial divergences and strict adversarial divergences, covering objectives from several GAN procedures and entropic regularized optimal transport.The strict subclass has the target distribution as its unique minimizer.Source: §1 · p. 2

Discriminator restrictions

Restricted discriminators produce generalized moment matching under certain conditions; linear f-GANs match feature expectations regardless of the chosen f-divergence.Neural-network GANs are characterized as suprema of linear GANs and share the moment-matching effect.Source: §1 · pp. 2–3

Convergence

For strict adversarial divergences, convergence in the objective implies weak convergence of the generated distribution to the target.This extends earlier convergence results for isolated objectives to a broad class of GANs.Source: §1 · p. 3

2 Notations

The notation section establishes conventions for functions, measures, topological spaces, and quotient spaces used in the paper.

Functions and measures

Function composition is written f ◦ g, function spaces as Y^X, product measures as µ ⊗ ν, and integration against µ as E_µ[f].Bold constants denote constant functions.Source: §2 · p. 3

Convex analysis

For a convex function f, dom f is its effective domain and f* denotes its convex conjugate.The conjugate is defined by f*(x*) = sup_x {x*·x − f(x)}.Source: §2 · p. 3

Spaces

The paper uses C(Ω), Cb(Ω), rca(Ω), and P(Ω) for continuous functions, bounded continuous functions, finite signed regular Borel measures, and probability measures.For a subspace Y of X, X/Y denotes the quotient space under the stated equivalence relation.Source: §2 · p. 3

3 General Framework

The framework treats GAN objectives as adversarial divergences, studies their minimizers, and sets up questions about approximation under restricted discriminators and convergence to those minimizers.

Interpretation

The framework distinguishes the objective’s minimizers from the objective itself, which is viewed as measuring the distance between the target and current estimate.The stated questions concern discriminator restriction and whether divergence convergence yields a standard distributional convergence.Source: §3 · pp. 4, 6

Adversarial divergences

An adversarial divergence is built from a function class and, with a fixed target distribution, reduces to the GAN objective; it is convex in each distribution argument.The framework captures objective functions used by multiple GAN-style procedures.Source: §3 · p. 4

Examples

Examples of the framework include MMD-GAN, WGAN, WGAN-GP, f-GAN, and optimal transport formulations.The cited optimal-transport formulations include regularized and unregularized forms; the paper notes neither was used in a GAN algorithm to its knowledge.Source: §3 · pp. 4–5

Minimizers

The set optτ,µ* collects probability measures minimizing τ(µ*||·), identifying the distributions against which approximation is assessed.The framework asks how these minimizers approximate the target under restricted discriminators.Source: §3 · pp. 5–6

Strict divergences

A strict adversarial divergence has the target as its unique minimizer; on compact metric spaces, MMD and Wasserstein examples induce metrics and are strict.The paper uses strictness to frame whether divergence convergence implies convergence to the minimizing set.Source: §3 · p. 6

4 Generalized Moment Matching

Section 4 relates restricted discriminator classes to generalized moment matching. It develops sufficient and stronger characterization conditions, then illustrates them for linear and neural-network f-GANs.

4 Generalized Moment Matching

For mθ(x, y) = vθ(x) − vθ(y), matching means equal expectations of vθ; Theorem 4 gives sufficient conditions for matched distributions to be optimal.Source: §4 · pp. 6–7
Under additional conditions, matching fully characterizes the optimal distributions, though Theorem 5's requirements are counterintuitive, especially for f-GANs.Source: §4 · pp. 7–8

4.1 Example: Linear f-GAN

For linear f-GANs, the optimum set consists exactly of distributions whose feature expectations match the target, independent of the chosen f.Source: §4.1 · p. 8

4.2 Example: Neural Network f-GAN

With differentiable activations and a linear final layer, neural-network f-GAN minimizers are exactly distributions indistinguishable by expectations of any discriminator network.Source: §4.2 · p. 9

5 Convergence

Section 5 defines weak convergence to sets of measures and proves that approaching an adversarial divergence's minimum forces weak convergence to its minimizer set on compact metric spaces. For strict divergences this is convergence to the target, enabling a hierarchy that places Wasserstein among the weakest strict divergences.

5 Convergence

Weak convergence reflects sample-space topology: δ1/n converges to δ0, but not to δ1.Source: §5 · pp. 9–10
The paper defines weak convergence to a set through convergence in the quotient space P(X)/A.Source: §5 · p. 10
On compact metric spaces, approaching an adversarial divergence's global minimum implies weak convergence to the set of its minimizers.Source: §5 · p. 10
For strict adversarial divergences, the minimizer set is the target alone, so objective convergence implies weak convergence to the target; this applies to f-GANs.Source: §5 · p. 10
The Wasserstein distance lies in the equivalence class of the weakest strict adversarial divergences because it metrizes weak convergence.Source: §5 · p. 11
For bounded closed Euclidean domains, trivial divergences occupy the strongest class while MMD and Wasserstein occupy the weakest; Figure 1 depicts equivalence classes.Source: §5 · pp. 11–12, Figure 1 · p. 12

6 Related Work

Prior work proposed GAN objectives based on f-divergences and MMD, while other studies examined convergence for specific objectives or in general IPM settings. This work characterizes a broader family and its convergence properties, but does not compare every pair of strict adversarial divergences.

6 Related Work

The paper identifies a general family of GAN-style objectives and establishes properties that apply across the family.Earlier examples include f-divergence objectives [10] and MMD objectives [5] [9].Source: §6 · p. 12
For strict adversarial divergences, the results generalize prior comparisons by showing they are stronger than Wasserstein distance and its equivalences.The passage contrasts this class-wide result with earlier results for specific divergences.Source: §6 · p. 12
The results do not compare two strict adversarial divergences when neither is equivalent to Wasserstein distance or the trivial divergence.Other cited theoretical work addresses generalization bounds and kernel-distance estimating power [3] [13] [5].Source: §6 · pp. 12–13

A Proof of Theorem 4

The proof invokes the theorem's assumptions for distributions in P(X) and uses equality of the specified expectations to describe membership in M.

A Proof of Theorem 4

The proof begins by applying the theorem's assumptions to an arbitrary µ ∈ P(X).The passage does not state the resulting property.Source: A Proof of Theorem 4 · p. 14
Membership of µ in M_{µ*} is specified by E_{µ*}[v_θ] = E_µ[v_θ] for every θ ∈ Θ.The proof fixes µ* and µ satisfying this equality.Source: A Proof of Theorem 4 · p. 14
The proof then proceeds from the moment-equality condition for the fixed distributions µ* and µ.The supplied passage ends before stating the next deduction.Source: A Proof of Theorem 4 · p. 14

B Proof of Theorem 5

The proof uses Theorem 4's inclusion of moment-matching distributions among the optimizers, then seeks the reverse inclusion by ruling out distributions with a nonzero m_{θ′} expectation.

B Proof of Theorem 5

Since Theorem 4 gives M_{µ*} ⊂ opt_{τ,µ*}, the proof reduces to establishing the reverse inclusion.The passage introduces this as the remaining proof obligation.Source: B Proof of Theorem 5 · p. 15
The proof considers a distribution µ for which some θ′ has E_{µ*⊗µ}[m_{θ′}] ≠ 0.It then derives a positive quantity before concluding that µ is not optimal.Source: B Proof of Theorem 5 · p. 15
For a minimizer, the proof invokes Fermat’s stationary-points theorem and uses the resulting gradient condition in its argument.The supplied proof fragments also state a nonzero expectation involving m − r and invoke the theorem again.Source: B Proof of Theorem 5 · p. 15

C Proof of Corollary 6

The corollary proof expresses the divergence using feature-based functions and establishes that its zero set corresponds to matching feature expectations.

C Proof of Corollary 6

The proof uses f*(x) ≥ x with equality at an interior point x0 to define a nonnegative penalty r_θ.It also invokes compactness of X and continuity of ψ when establishing the relevant differentiability conditions.Source: C Proof of Corollary 6 · pp. 15–16
The proof writes τ(µ||ν) as a supremum of E_{µ⊗ν}[m_θ − r_θ], with v_θ and r_θ bounded and continuous under its stated assumptions.Here m_θ compares v_θ at x and y, while r_θ is defined from f* and the feature map ψ.Source: C Proof of Corollary 6 · p. 16
The proof states that optτ,µ* is the set of distributions µ with τ(µ*||µ)=0 and includes µ* in that set, while noting that an additional equality remains to be shown.Source: C Proof of Corollary 6 · p. 16
When the feature expectations match, the proof obtains E_{µ⊗ν}[m_θ] = 0 for every θ ∈ Θ.The passage identifies this as verifying condition (6).Source: C Proof of Corollary 6 · p. 16

D Proof of Theorem 10

The proof uses weak-* compactness and lower semicontinuity to establish that minimizers exist, then shows sequences approaching the infimum converge to the minimizer class in the quotient space.

D Proof of Theorem 10

Lemma 13 establishes weak-* compactness of P(X) from compactness of a measure ball and the weak-* closed probability constraints.The proof uses Banach–Alaoglu and the characterization of total mass one as a closed condition.Source: D Proof of Theorem 10 · pp. 16–17
Lower semicontinuity of τ(µ∗||·) on compact P(X) ensures the minimum is attained, so optτ,µ∗ is nonempty.The divergence is a supremum of affine continuous functions, making it lower semicontinuous.Source: D Proof of Theorem 10 · p. 17
In the quotient by optτ,µ∗, compactness of the complement of any minimizer neighborhood and lower semicontinuity show that near-minimizing sequences eventually enter that neighborhood.Outside the neighborhood the divergence attains a value strictly above its infimum; convergence to the infimum then forces eventual entry.Source: D Proof of Theorem 10 · p. 17

E Proof of Corollary 12

The proof compares divergence strengths and their induced convergence: KL variants are strictly stronger than total variation, while Wasserstein and MMD metrize weak-* convergence on compact metric spaces. The trivial divergence is stronger than every strict adversarial divergence.

E Proof of Corollary 12

Total variation is equivalent to JS divergence but not to Wasserstein distance.The passage also relates total variation and JS to squared Hellinger distance.Source: E Proof of Corollary 12 · p. 18
KL and reverse-KL are strictly stronger than total variation, yet the two KL variants are not comparable with each other and are weaker than the trivial divergence.Shifted uniform distributions give examples where both KL values remain infinite while total variation tends to zero.Source: E Proof of Corollary 12 · p. 18
Wasserstein and MMD metrize weak-* convergence on compact metric spaces, placing them in the equivalence class of the weakest strict adversarial divergences.The passage attributes the metrizability result to [12] and invokes Theorem 10 for the equivalence-class conclusion.Source: E Proof of Corollary 12 · p. 18
The trivial divergence is stronger than any strict adversarial divergence because convergence under it makes a sequence eventually equal to the target.Source: E Proof of Corollary 12 · p. 18