On Self Modulation for Generative Adversarial Networks

1 INTRODUCTION

• GAN training is difficult because it searches for a Nash equilibrium in a high-dimensional, non-convex game, while label-based conditioning requires external information that may be unavailable.The paper identifies stabilizing GAN training as a major research challenge.Source: §1 · p. 1
• Self-modulation conditions generator layers on the generator's own input rather than external information, and is presented as a simple change applicable to popular generator architectures.The approach is motivated by Feature-wise Linear Modulation in supervised learning.Source: §1 · p. 1
• Self-modulation improves sample quality by 5%-35% and outperforms the baseline in 124/144 studied settings.The settings span loss functions, regularizers and normalizers, architectures, and optimization settings.Source: §1 · pp. 1–2

2 SELF-MODULATION FOR GENERATIVE ADVERSARIAL NETWORKS

• Self-modulation makes hidden activations depend feature-wise on latent vector z, allowing generator feature maps to be re-weighted as a function of the input.The proposal is motivated by FiLM, which conditions supervised networks on side information.Source: §2 · p. 2
• For unconditional generation, the method replaces fixed batch-normalization scale and shift parameters with input-dependent β(z) and γ(z), parameterized by a neural network of z.The implementation uses a small one-hidden-layer ReLU MLP; β uses independent parameters.Source: §2 · pp. 2–3
• Side information can be incorporated by composing it with z into z′, which can then be used directly in the self-modulation equation.The paper's implementation uses a bilinear interaction between z and two trainable label-embedding functions.Source: §2 · p. 3
• The authors report that self-modulation outperforms standard conditional models in the settings they demonstrate.Source: §2 · p. 3
• The method is presented through batch normalization, but the normalization component can be removed so modulation operates independently.Batch normalization is used as a pragmatic choice because it is common in neural networks.Source: §2 · p. 3
• More complex modulation architectures, including attention-based modulation and a constrained γ gating mechanism, did not substantially increase performance in initial experiments.Source: §2 · p. 3

3.1 EXPERIMENTAL SETTINGS

• The robustness study crossed two losses, two architectures, and regularization and hyperparameter settings across four datasets, yielding 1,440 trained models over five seeds per method.There were 36 settings per dataset, with medians computed across random seeds.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 5

3.2 ROBUSTNESS EXPERIMENTS FOR UNCONDITIONAL GENERATION

• Self-modulation outperformed the baseline in 124/144 paired comparisons, with all other settings held fixed.The paired setting includes optimization hyperparameters in the model tuple, so the comparison does not require additional hyperparameter tuning.Source: §3.2 · p. 5
• Across unpaired settings, FID decreased by 4.3%-33%; self-modulation won in all RESNET settings and 87.5% of SNDCGAN cases.The SNDCGAN exceptions were two settings on LSUN-BEDROOM.Source: Table 2 · p. 5, Table 3 · p. 6
• In the considered label-conditional settings, incorporating self-modulation significantly improved performance.Source: §3.2 · p. 6, Table 4 · p. 7
• At 500k generator steps, unconditional FID was 60.4 for the baseline and 53.7 for self-modulation (11% improvement), using one setting and three seeds.The authors note that this run required about 10 days on a P100 GPU.Source: §3.2 · p. 6
• FID improved from 50.6 to 43.9 (13% improvement); IS improved from 14.1 to 15.1 unconditionally and from 20.1 to 22.2 conditionally.Source: §3.2 · p. 7
• Modulation benefits are greatest in the last layer, but applying it at every layer gives the best performance; the pattern persists across datasets.Source: Figure 2 · p. 6, §3.2 · p. 7

4 RELATED WORK

• Conditional GANs inject side information by concatenating it with the input vector or by using conditional batch normalization in the generator.Self-modulation is presented as complementary, especially when side information is unavailable, and applicable when it is available.Source: §4 · p. 7
• Conditional modulation uses side information to modulate neural-network computation, while the proposed method also works without side information.The related-work discussion connects this family to conditional instance normalization, dynamic layer normalization, and FiLM.Source: §4 · p. 7
• Batch-normalization modulation combines multiplicative and additive effects, linked respectively to gating and residual networks.The proposed method adopts both types of modulation.Source: §4 · p. 7

5 DISCUSSION

• Self-modulation is presented as a generator modification that improves the performance of most GANs.The authors describe it as simple to implement and applicable to popular GANs.Source: §5 · p. 7
• The results show performance gains but do not explain how self-modulation produces better models.The authors note that GAN training is less well understood, motivating diagnostic analysis rather than speculation.Source: §5 · pp. 7–8
• Figure 3 relates generator Jacobian condition number to FID and reports correlations of 0.67 and 0.83 on CIFAR10 and 0.24 and 0.39 on IMAGENET for Self-Mod and Base, respectively.The condition number is computed from minibatch Jacobians, with log condition numbers averaged across examples.Source: Figure 3 · p. 8, §5 · p. 8
• Self-modulated models tend to favor higher recall, most notably on IMAGENET.The precision/recall measures interpret producing low-probability data points as a precision loss and failing to generate high-probability points as a recall loss.Source: §5 · pp. 8–9
• The diagnostics suggest that self-modulation stabilizes the generator toward favorable conditioning values and may improve mode coverage.The discussion cautions that better tools and theory are needed to disentangle symptoms from causes.Source: §5 · p. 9

A.1 INCEPTION SCORES

• Table 5 reports the median score across random seeds and the best attainable score across optimization hyperparameters in the unpaired setting, comparing SELF-MOD with the batch-normalization baseline.The table caption identifies the unpaired setting as defined in Section 3.2.Source: Table 5 · p. 12

A.2 FIDS

• Table 6 gives the standard error of the median for Table 2.Source: Table 6 · p. 12

A.3 WHICH LAYER TO MODULATE?

• Figure 4 presents performance when different generator layers are modulated for each dataset.Source: §A.3 · p. 13, Figure 4 · p. 13

A.4 CONDITIONING AND PRECISION/RECALL

• Figure 5 presents generator Jacobian condition number and precision/recall plots for each dataset.Source: §A.4 · p. 13

B MODEL ARCHITECTURES

• The section describes the model structures used in the experiments.Source: B MODEL ARCHITECTURES · p. 13

• SNDCGAN follows the architecture used in Miyato et al. (2018).Source: §B.1 · p. 13
• The work uses different SNDCGAN architectures for 32×32×3 CIFAR10 images and 128×128×3 images in other datasets.Source: §B.1 · p. 13
• Self-modulation replaces the generator's existing BN layer and is abbreviated sBN in the tables.Source: §B.1 · p. 13

• The ResNet architecture follows the one used in Miyato et al. (2018).Source: §B.2 · p. 13
• Two ResNet architectures are used because image resolutions differ.Source: §B.2 · p. 13
• Self-modulation replaces the existing BN layer and is abbreviated sBN in the tables.Source: §B.2 · p. 13

• Generator conditional batch normalization uses label information, with each label associated with its own scaling and shifting parameters.Source: §B.3 · p. 13
• Projection-based discriminator conditioning adds the dot product of the final feature representation and label embedding to the output logits.The passage describes this as encouraging use of label-discriminative features to distinguish real from fake samples.Source: §B.3 · p. 13
• Figure 5 plots generator log condition number against FID in the left-hand plots and generator precision/recall metrics in the right-hand plots.Each plotted point represents one model, selected by mean FID across five random seeds.Source: Figure 5 · p. 14
• The comparison uses generator-only label conditioning and full P-cGAN conditioning in both generator and discriminator.The setup follows Miyato & Koyama (2018), with two discriminator steps per generator step instead of five.Source: §B.3 · pp. 14–15
• SNDCGAN generator and discriminator tables cover 32×32×3 and 128×128×3 resolutions.Source: Table 7 · p. 15, Table 8 · p. 16, Table 9 · p. 16
• ResNet generator and discriminator tables cover both resolutions; generator blocks use upsampling skip connections, while discriminator blocks use 1×1 convolution skip connections with possible downsampling.Source: Table 11 · p. 17, Table 12 · p. 17, Table 13 · p. 18