Source-linked AI summary
Banach Wasserstein GAN
Jonas Adler, Sebastian Lunz
TL;DR
WGANs traditionally use ℓ2 image distances, limiting how generated-image features are emphasized. The paper extends gradient-penalty WGANs to separable Banach spaces and studies norm choices including Sobolev spaces. BWGAN changes performance across CIFAR-10 and CelebA, with Space L10 reaching an 8.31 unsupervised inception score on CIFAR-10.
Problem
WGANs have used ℓ2 as the underlying image distance, despite other metrics emphasizing features such as edges, global structure, or perceptual similarity.
Method
The paper extends WGANs with gradient norm penalization to separable Banach spaces and implements arbitrary underlying norms through their dual norms.
Results
8.31 unsupervised inception score on CIFAR-10 is achieved by BWGAN using Space L10, reported as state of the art for non-progressive growing GANs.
Takeaways & Limitations
Choosing the underlying norm provides a design degree of freedom for emphasizing image features relevant to a specific application.
Takeaways & Limitations
Difference-quotient penalization may fail to effectively enforce the Lipschitz condition when distributions are already close, while gradient norm penalization addresses all spatial directions through the dual norm.
Abstract
from arXiv · showhide
Wasserstein Generative Adversarial Networks (WGANs) can be used to generate realistic samples from complicated image distributions. The Wasserstein metric used in WGANs is based on a notion of distance between individual images, which induces a notion of distance between probability distributions of images. So far the community has considered $\ell^2$ as the underlying distance. We generalize the theory of WGAN with gradient penalty to Banach spaces, allowing practitioners to select the features to emphasize in the generator. We further discuss the effect of some particular choices of underlying norms, focusing on Sobolev norms. Finally, we demonstrate a boost in performance for an appropriate choice of norm on CIFAR-10 and CelebA.
1 Introduction
BWGAN extends WGAN gradient-penalty theory beyond the conventional ℓ2 image distance, enabling norms that emphasize selected image features. Experiments compare such norms on CIFAR-10 and CelebA, including an 8.31 inception score on CIFAR-10.
- WGANs use a generator to map random noise to samples and a critic to distinguish generated from real images.
- Existing WGAN implementations use ℓ2 as the underlying image distance, although imaging metrics can emphasize features such as edges or human perceptual similarity.
- The paper extends WGAN gradient-penalty theory to Banach spaces, allowing norms such as Sobolev norms to emphasize edges or large-scale behavior.
- BWGAN replaces the classical ℓ2 norm with a dual norm in practice and provides heuristics for choosing regularization parameters.
- 8.31 unsupervised inception score on CIFAR-10 is achieved with Space L10, reported as state of the art for non-progressive growing GANs.
2 Background
The background develops WGANs from distributional divergences to Wasserstein distances and explains why the underlying image norm matters. It motivates Banach-space and Sobolev formulations, whose norms can emphasize outliers, edges, global structure, or high-frequency detail.
- GAN training maps latent noise to generated images and uses a discriminator to distinguish them from real images.
- The Wasserstein metric remains finite and provides meaningful generator gradients for mutually singular distributions by measuring transport cost rather than point-wise divergence.
- The Wasserstein distance depends strongly on the underlying image metric, so changing the norm changes which image differences are emphasized.
- Gradient norm penalization enforces the Lipschitz condition through the dual norm, whereas difference-quotient penalization may constrain only the sampled data direction.
- WGAN gradient penalty is specifically tied to ℓ2 and does not directly extend to arbitrary underlying spaces; the paper generalizes it to separable Banach spaces.
- Higher p in Lp spaces increases emphasis on outliers, while Sobolev norms can emphasize edges and control image frequencies through s.
3 Banach Wasserstein GANs
The paper extends gradient-penalty WGANs from ℓ2 image spaces to separable Banach spaces by enforcing Lipschitzness through dual norms. It derives implementation and parameter-selection guidance that supports training with arbitrary underlying norms.
- BWGAN generalizes gradient-penalty WGANs to any separable complete normed space.
- In Banach spaces, a function is γ-Lipschitz exactly when the dual norm of its Fréchet derivative is at most γ everywhere.
- The Banach-space gradient-penalty loss uses the dual norm of the critic derivative, and its minimizer approximates the Wasserstein distance on B.
- Finite-dimensional implementations can use standard automatic differentiation, with the Banach-space and adjoint operators handled implicitly.
- Changing the underlying norm affects computational cost primarily through the gradient of the dual norm, while network forward and backpropagation remain unchanged.
- 3.2 Regularization parameter choices: The regularization parameters λ and γ are norm-dependent, so the paper derives heuristics to avoid hand-tuning them for each norm.
- 3.2 Regularization parameter choices: For CIFAR-10 with ℓ2, the heuristic gives λ ≈27 and γ = λ ≈27.
4 Computational results
BWGAN was evaluated with Sobolev and Lp norms on CIFAR-10 and CelebA, showing that norm choice affects image quality and training stability. On CIFAR-10, larger p and negative s performed better, while CelebA favored s between −1 and 0 and around p = 0.
- Experimental setup: BWGAN was implemented with various Sobolev and Lp norms on 64 × 64 CIFAR-10 and CelebA images.The implementation used the residual architecture from [7] and replaced the classical norm according to the selected Sobolev or Lp space.
- Evaluation: FID and Inception scores were computed from 50K samples, with higher Inception and lower FID indicating better image quality.
- CIFAR-10: On CIFAR-10, both FID and Inception scores indicated that negative s and large p produced better image quality.
- CelebA: On CelebA, the best FID scores occurred for s between −1 and 0 and around p = 0, whereas training became unstable for p = 10.
- CIFAR-10: 8.31 ± 0.07 was the Inception score achieved by L10-BWGAN, reported as state of the art among non-progressive growing methods.
- CIFAR-10: 16.43 was the CIFAR-10 FID achieved using L4, while W 0,2 = ℓ2 slightly exceeded the reference implementation.
5 How about metric spaces?
The paper considers extending Wasserstein GAN regularization beyond Banach spaces to general metric spaces through direct Lipschitz enforcement. It argues that difference-quotient penalization can fail to enforce the full Lipschitz constraint when sampled points are close, while gradient norm penalization controls all directions.
- Metric-space extension: Directly enforcing Lipschitz continuity could extend Wasserstein GAN training to general metric spaces beyond Banach spaces.
- Limitations: Difference-quotient penalization may have considerably higher variance in practice despite theoretical equivalence under appropriately chosen distributions.
- Limitations: When sampled points are sufficiently close, the Lipschitz quotient constrains the differential only along the direction joining those points.
- Limitations: For close distributions, sampled directions may align with the data and leave the Lipschitz constraint insufficiently enforced.
- Comparison: Gradient norm penalization instead uses the dual norm of the differential to enforce the condition simultaneously in all spatial directions.
- Metric assumptions: When the metric is bounded below, the preceding failure argument does not apply; with the trivial metric, Wasserstein GAN approximates Total Variation distance.
6 Conclusion
The paper generalizes WGAN gradient norm penalization to Banach spaces, enabling norms that emphasize application-relevant image features. Experiments on CIFAR-10 and CelebA show that norm choice affects performance, with Sobolev parameters correlating with FID.
- Banach Wasserstein GAN generalizes WGAN gradient norm penalization from the ℓ2 setting to Banach spaces and diverse image norms.The theory is intended to support norms tailored to features relevant in specific applications.
- Changing the underlying norm affected model performance on CIFAR-10 and CelebA.
- FID scores across Sobolev spaces W s,p correlated with both s and p on CIFAR-10 and CelebA.
- The generalized framework applies beyond images to data in any normed space.
A Some further Banach spaces
This section presents constructions for forming useful Banach spaces from existing spaces, including weighted and product spaces. These constructions can emphasize selected image features or represent multiple data modalities.
- Weighted spaces: Weighted spaces define a new norm by applying a continuous linear bijection to elements of a separable Banach space.The construction can emphasize features such as a color channel or a spatial image region.
- Weighted spaces: The dual-space mapping for weighted spaces is the adjoint of the inverse of the defining operator.
- Product spaces: Product spaces combine Banach spaces B1 through Bn into a single product space.
- Product spaces: For product spaces, the dual norm uses conjugate exponents satisfying 1/p + 1/q = 1.Such spaces can model color channels or multimodal outputs such as an image paired with a caption.
B Network details
The CIFAR-10 and CelebA implementations use residual-network architectures adapted from the source implementation. The CelebA model adds residual blocks for further upsampling and downsampling.
- CIFAR-10: The CIFAR-10 generator starts from 128 normally distributed latent variables and repeatedly upsamples from 4x4 to 32x32 resolution.Its building blocks are residual blocks with nonlinearities, convolutions, and residual connections.
- CIFAR-10: The CIFAR-10 discriminator reverses the generator’s spatial progression using pooling, spatial mean-pooling, and a final dense layer.
- CIFAR-10: The implementation uses ReLU nonlinearities, 128 convolutional channels, generator batch normalization, and He or Xavier initialization.
- CelebA: The CelebA implementation follows CIFAR-10 while adding a residual block for further upsampling and downsampling in both networks.
- Further implementation details are available in the cited source and open-source implementation.
C Further samples
The appendix provides samples and Fréchet derivatives for the investigated Sobolev and Lp spaces on CIFAR-10 and CelebA. Higher Sobolev s corresponds to higher discriminator-derivative gradients and greater emphasis on high-frequency content.
- Sobolev spaces: Higher s in the investigated Sobolev spaces indicates higher gradients in the discriminator’s Fréchet derivative and emphasis on higher-frequency content.
- The appendix’s images were downsampled for arXiv size restrictions, with a full-resolution version available online.
- Sobolev spaces: Figure 7 presents CIFAR-10 samples for all investigated W s,2 spaces.
- Sobolev spaces: Figure 8 presents CIFAR-10 Fréchet derivatives for all investigated W s,2 spaces.
- Sobolev spaces: Figure 9 presents CelebA samples for all investigated W s,2 spaces.
- Sobolev spaces: Figure 10 presents CelebA Fréchet derivatives for all investigated W s,2 spaces.
- Lp spaces: Figures 11 and 12 present CIFAR-10 samples and Fréchet derivatives for all investigated Lp spaces.
- Lp spaces: Figures 13 and 14 present CelebA samples and Fréchet derivatives for all investigated Lp spaces.