Sobolev GAN

• Sobolev IPM restricts the critic to a Sobolev ball defined with respect to a dominant measure μ.Source: §1 · pp. 2–3
• Unlike Fisher IPM’s joint density comparison, Sobolev IPM compares weighted coordinate-wise conditional CDFs on a leave-one-out basis.The paper identifies conditional-dependency matching as crucial for sequence modeling.Source: §1 · p. 2
• The metric incorporates conditional-dependency modeling into GAN learning, giving Sobolev IPM an advantage over Fisher IPM for sequence comparison.Source: §1 · p. 2
• Sobolev IPM’s intrinsic conditioning supports discrete sequence and text generation, while Fisher IPM can succeed with recurrent critics, generators, and curriculum conditioning.Source: §1 · pp. 3–4
• The paper provides an ALM algorithm for training Sobolev GAN that is stable without compromising critic capacity.Source: §1 · p. 3
• A Sobolev GAN variant achieves competitive semi-supervised learning results on CIFAR-10 without critic activation normalization.The paper attributes this result to smoothness enforced by the Sobolev regularizer.Source: §1 · pp. 3–4

• IPMs compare distributions through the supremum of a mean discrepancy over a function class F, with closed forms sometimes expressed using PDFs, CDFs, or score functions.Source: §2 · p. 4
• Sobolev IPM compares weighted coordinate-wise conditional CDFs, whereas Fisher IPM compares joint PDFs, making the Sobolev metric conditioning-oriented.Source: §2 · p. 4
• The reviewed metrics are presented in Table 1 as alternatives for GAN training, with references to their uses in that setting.Source: §2 · p. 4, Table 1 · p. 5

• Generalized Fisher IPM relaxes data-independent constraints such as global Lipschitzness by constraining the critic in a Lebesgue ball defined with respect to a dominant measure μ.Source: §3 · p. 6
• The framework assumes μ is a positive distribution on X and defines L2(X, μ) as the space of μ-measurable functions with finite norm.Source: §3 · p. 6
• The Fisher distance equals the L2(X, μ) norm of the density difference, and its optimal critic is characterized by Theorem 1.Source: §3 · p. 7
• The dominant measure determines where the optimal critic and the weighted density comparison are evaluated, with the average taken over points sampled from μ.Source: §3 · p. 7
• Choosing μ = 1/2(P + Q) recovers the symmetric chi-squared distance, while other choices include interpolation measures and can yield nonsymmetric divergences.Source: §3 · p. 7

4 Sobolev IPM

• Sobolev IPM constrains the critic’s gradient in a restricted Sobolev space, producing a metric based on weighted coordinate-wise conditional CDFs rather than joint PDF matching.Source: §4 · p. 7

4.1 Definition and Expression of Sobolev IPM in terms of Coordinate Conditional CDFs

• Theorem 2 gives equivalent forms involving high-order partial derivatives of CDFs and weighted coordinate-wise conditional CDFs, under continuous differentiability assumptions.Source: §4.1 · pp. 8–9

4.2 Illustrative Examples

• The optimal Sobolev critic solves an elliptic PDE with zero boundary conditions, and its gradient defines a transportation plan on the support of μ.Source: §4.1 · p. 9, §4.2 · p. 11, Figure 1 · p. 11

4.1 Definition and Expression of Sobolev IPM in terms of Coordinate Conditional CDFs

• The dominant measure μ determines the support and weighting of the average conditional-CDF discrepancy.Source: §4.1 · p. 9
• For each coordinate, the metric compares its conditional CDF given all other variables, weighted by the density of those other variables on a leave-one-out basis.Source: §4.1 · p. 10

4.2 Illustrative Examples

• In one dimension, Sobolev IPM becomes the Cramér distance for uniform μ, comparing CDFs rather than the inverse CDFs compared by Wasserstein-p.Source: §4.2 · p. 10
• For point masses, CDF comparison is more suitable than PDF comparison on discrete spaces.Source: §4.2 · p. 10

• Sobolev GAN samples real, latent, and dominant-measure points to optimize a generator whose distribution approaches the real data distribution.The dominant measure may be the real-plus-fake mixture or the interpolation distribution used by WGAN-GP.Source: §5 · pp. 11–12
• Training alternates augmented-Lagrangian critic optimization with unconstrained generator updates based on the critic’s mean discrepancy.The critic uses a sphere constraint and updates the Lagrange multiplier during optimization.Source: §5 · p. 12
• The algorithm samples minibatches, updates critic and multiplier parameters with ADAM and a multiplier learning rule, then updates the generator by stochastic gradient descent.Source: §5 · p. 12
• The Sobolev objective imposes an average gradient-norm constraint under the dominant measure, rather than WGAN-GP’s pointwise quadratic gradient penalty.This average constraint enables the coordinate-weighted conditional CDF interpretation.Source: §5 · p. 13

6.1 Text Generation with Sobolev GAN

• The text-generation study evaluates loss, critic and generator architecture, and dominant-measure choice using JS-4 on 32-character sequences from the Google Billion Word dataset.Real-distribution Gaussian smoothing is annealed during training to accommodate discrete sequences.Source: §6.1 · pp. 13–14
• Sobolev GAN with annealed noise smoothing achieves slightly better JS-4 results than WGAN-GP when the initial noise level is σ0 = 1.5.Without smoothing and annealing, Sobolev GAN is behind; annealing gradually changes the real distribution from continuous toward discrete.Source: §6.1 · p. 14
• Fisher GAN fails with a ResNet generator and several critic architectures, whereas recurrent critics improve marginally and curriculum-conditioned recurrent training reaches Sobolev GAN and WGAN-GP levels.The comparison highlights Sobolev GAN’s implicit conditioning through its gradient regularizer.Source: Figure 3 · p. 15, §6.1 · p. 15

6.2 Semi-Supervised Learning with Sobolev GAN

• A variant of Sobolev GAN achieves competitive CIFAR-10 semi-supervised learning results without batch, layer, or weight normalization in the critic.The reported setup uses the K + 1 critic formulation and compares error rates across different numbers of labeled samples.Source: §6.2 · pp. 16–18, Table 2 · p. 18
• The semi-supervised model shares a CNN between classifier and critic, with K classifier directions defining the real component and a K + 1th direction identifying fake samples.The authors report better results for this K + 1 formulation than for a formulation where the classifier layer does not interact with the critic direction.Source: §6.2 · pp. 16–17, Figure 4 · p. 16
• Applying Sobolev regularization only to the fake critic preserves labeled-set fitting while enforcing smoothness on the shared feature representation.The Fisher constraint stabilizes training, and the Sobolev constraint on f− is related to Laplacian regularization.Source: §6.2 · p. 17

7 Conclusion

• Sobolev GAN achieves competitive semi-supervised learning results without normalization, attributed to smoothness induced by its gradient regularizer.Source: §7 · p. 18
• Sobolev IPM compares weighted coordinate-wise CDFs and provides intrinsic conditioning useful for structured and discrete data.Source: §7 · p. 18
• Both Sobolev IPM's implicit conditioning and Fisher IPM's explicit conditioning succeed in text generation, with different tradeoffs.Source: §7 · p. 18

A Theory: Approximation and Transport Interpretation

• The theoretical section relates Sobolev IPM to distribution transport and other distribution metrics, including the Stein distance.Source: A Theory: Approximation and Transport… · p. 23

A.1 Approximating Sobolev IPM in a Hypothesis class

• The full Sobolev space is intractable, so optimization is restricted to a hypothesis class such as neural networks whose functions vanish at the boundary.Source: §A.1 · p. 23
• Sobolev IPM approximation error is proportional to how well the hypothesis class approximates the optimal Sobolev critic in the Sobolev norm.Source: §A.1 · p. 23
• The hypothesis class must have enough representation power to express the gradient of the optimal critic, paralleling Fisher IPM approximation in the Lebesgue norm.Source: §A.1 · p. 23

A.2 Distribution Transport Perspective on Sobolev IPM

• The optimal Sobolev critic satisfies a nonlinear PDE derived from the variational problem.Source: §A.2 · pp. 23–24
• The analysis links Sobolev and Stein discrepancies through an inequality relating Sobolev distance to Stein goodness of fit.Source: §A.2 · p. 24
• The PDE can be related to the Fokker–Planck equation by identifying the gradient of the Sobolev critic as a drift.Source: §A.2 · p. 24
• Sobolev descent evolves particles along the critic gradient, and the limiting particle distribution is the target distribution.Source: §A.2 · pp. 24–25
• The critic gradient therefore defines a transportation plan for moving generator mass from Q to P during GAN training.Source: §A.2 · p. 25

B Proofs

• The approximation proof shows that Sobolev critic error is measured in the Sobolev seminorm, which is stronger than the Lebesgue norm under the Poincaré inequality.Source: B Proofs · pp. 26–27
• Calculus of variations and KKT optimality conditions yield the PDE satisfied by the optimal Sobolev critic under boundary-vanishing assumptions.Source: B Proofs · pp. 27–28
• The proof identifies the Sobolev critic PDE with a Stein-operator formulation and derives an inequality involving Sobolev distance and Stein discrepancy.Source: B Proofs · pp. 30–31
• If Sobolev distance S_P(P, Q_n) tends to zero, the density ratio Q_n(x)/P(x) converges in expectation toward one, with speed governed by Stein discrepancy.Source: B Proofs · p. 31
• The resulting density-ratio convergence is weaker than convergence controlled by conditional CDFs through Sobolev distance and by score-function fitness through Stein discrepancy.Source: B Proofs · p. 31

Text experiments: Additional Plots

• Annealed smoothing of P_r outperforms non-annealed smoothing in Sobolev GAN experiments.The paper describes annealing as gradually moving the real distribution from continuous to discrete.Source: Figure 5 · p. 32
• Using an RNN critic degrades WGAN-GP and Sobolev performance when the generator architecture is fixed to ResNets.The experiments use μ = μ_GP and apply gradient clipping without other stabilization tricks.Source: C Text experiments: Additional Plots · p. 32, Figure 6 · p. 32
• Figure 7 provides text samples from the various GANs considered in the paper.Source: Figure 7 · p. 33

D SSL: hyperparameters and architecture

• CIFAR-10 models train for 350 epochs with Adam at learning rate η = 2e−4, β1 = 0.5, and β2 = 0.999.The setup also uses selected cross-entropy and weight-decay coefficients, with distinct settings for Fisher-only and Sobolev-plus-Fisher formulations.Source: D SSL: hyperparameters and architecture · p. 33
• The generator is a 32×32 DCGAN with transposed convolutions, batch normalization, ReLU activations, and a final Tanh output.It uses G_extra_layers=2 and begins from a 100-dimensional input.Source: D SSL: hyperparameters and architecture · p. 33
• The critic uses dropout, convolutional LeakyReLU blocks, and separate linear heads V: 6144 -> 1 and S: 6144 -> 10.The architecture combines a scalar output with a ten-class output for the semi-supervised setup.Source: D SSL: hyperparameters and architecture · pp. 33–34