Source-linked AI summary
Self-Supervised GANs via Auxiliary Rotation Loss
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, Neil Houlsby
TL;DR
Conditional GANs perform well but require labeled data, while GAN training can make discriminator representations unstable and forgetful. This paper adds rotation-based self-supervision to adversarial training, making the generator and discriminator collaborative for representation learning. The resulting model matches conditional GAN performance under comparable conditions and reaches an FID of 23.4 for unconditional ImageNet generation.
Problem
Conditional GANs benefit from labeled data, but GAN training exposes discriminators to non-stationary distributions and representation forgetting.
Method
The self-supervised GAN adds an auxiliary rotation-prediction loss to the discriminator while retaining adversarial true-versus-fake training.
Results
The model matches equivalent conditional GANs on image synthesis and attains an FID of 23.4 on unconditional ImageNet generation.
Takeaways & Limitations
Adversarial and self-supervised learning can be combined for fully unsupervised natural image synthesis with stable representations and competitive image quality.
Takeaways & Limitations
Future work includes using state-of-the-art self-supervised discriminator architectures and adding techniques such as self-attention and orthogonal normalization.
Abstract
from arXiv · showhide
Conditional GANs are at the forefront of natural image synthesis. The main drawback of such models is the necessity for labeled data. In this work we exploit two popular unsupervised learning techniques, adversarial training and self-supervision, and take a step towards bridging the gap between conditional and unconditional GANs. In particular, we allow the networks to collaborate on the task of representation learning, while being adversarial with respect to the classic GAN game. The role of self-supervision is to encourage the discriminator to learn meaningful feature representations which are not forgotten during training. We test empirically both the quality of the learned image representations, and the quality of the synthesized images. Under the same conditions, the self-supervised GAN attains a similar performance to state-of-the-art conditional counterparts. Finally, we show that this approach to fully unsupervised learning can be scaled to attain an FID of 23.4 on unconditional ImageNet generation.
1. Introduction
GANs can synthesize data without labels but face unstable training and discriminator forgetting. The self-supervised GAN combines adversarial and self-supervised learning to recover benefits associated with conditional models without labeled data.
- GAN training is unstable because the generator and discriminator seek a Nash equilibrium in a high-dimensional, non-convex game.
- As generated samples change, the discriminator operates in a non-stationary environment and may forget previous classification boundaries.
- Conditional GANs provide labeled information that encourages more stable representations, but require labeled data.
- The self-supervised GAN adds an auxiliary loss so the generator and discriminator collaborate on representation learning while competing on generation.
- Under the same training conditions, the model closes the natural-image-synthesis gap between unconditional and conditional models.
2. A Key Issue: Discriminator Forgetting
GAN training creates a non-stationary classification problem in which discriminator representations can be forgotten. Evidence from sequential classification and GAN training shows that self-supervision helps retain class information.
- As the generator changes, the discriminator faces a non-stationary online learning problem.
- The discriminator may forget useful representations because changing tasks emphasize different levels of structure, texture, and detail.
- At convergence, a perfect generator makes the optimal discriminator output 0.5 independently of the input, removing its requirement to retain meaningful representations.
- In sequential CIFAR10 classification, accuracy drops substantially after each task switch and no useful information carries across the task cycle.
- During unconditional GAN training, ImageNet classification accuracy first increases and later decreases, indicating that discriminator class information is acquired and forgotten.
3. The Self-Supervised GAN
The self-supervised GAN augments the discriminator with rotation prediction alongside true-versus-fake discrimination. The generator and discriminator cooperate on rotation-based representation learning while remaining adversarial on generation.
- Self-supervision trains the discriminator on an auxiliary pretext task, such as predicting an image’s rotation angle.
- Self-supervised loss helps representations transfer across sequential tasks, with performance continually improving in the illustrated setting.
- A single discriminator uses two heads for true-versus-fake prediction and rotation classification.
- The generator produces upright images whose rotated versions should remain suitable for discriminator rotation detection.
4. Experiments
Experiments show that self-supervision improves unconditional GAN stability, sample quality, and discriminator representations, often matching conditional GAN performance under comparable conditions.
- Sample Quality: FID results show SS-GAN substantially outperforms Uncond-GAN on IMAGENET and matches Cond-GAN when self-modulation is added.The best SS-GAN reaches an FID of 23.4 on unconditional IMAGENET generation.
- Sample Quality: On CIFAR10 and LSUN-BEDROOM, SS-GAN substantially improves over Uncond-GAN while matching conditional GAN performance.Self-supervision does not significantly improve results on CELEBA-HQ, possibly because its low diversity makes rotation less informative.
- Training Stability: Adding self-supervision reduces unconditional GAN sensitivity to hyperparameters and prevents collapses that produce FIDs larger than 100.Mean FID scores are compared across penalties and optimization hyperparameters.
- Representation Quality: SS-GAN produces the best representations across IMAGENET blocks and training iterations, with similar results on CIFAR10.The representation quality of Uncond-GAN drops at 500k steps, consistent with its FID decline.
- Ablation: Ablating the adversarial loss substantially decreases representation quality, indicating that adversarial and rotation losses complement each other.The discriminator architecture is optimized for image generation rather than representation quality.
- Representation Quality: SS-GAN representations are competitive with state-of-the-art self-supervised methods and outperform BiGAN by 0.073 accuracy points.DeepClustering performs only 0.027 accuracy points better but requires offline clustering after every training epoch.
5. Related Work
The related work covers GAN forgetting and conditioning, then contrasts label-dependent conditional GANs with self-supervised representation learning without labels.
- GAN forgetting: GAN training instability has been linked to discriminator forgetting in a non-stationary environment.Proposed remedies include replaying generated images, averaging checkpoints, and maintaining queues of previous models.
- Representation evaluation: Table 5 compares self-supervised representation-learning methods using top-1 accuracy on ImageNet, reporting mean performance for SS-GAN.The supplied table passage specifies the evaluation metric and aggregation for SS-GAN but not the numerical values.
- Conditional GANs: Conditional GANs improve generative modeling of complex datasets through label-based conditioning and auxiliary discriminator objectives.AC-GAN and P-cGAN are described as key examples of this approach.
- Comparison: The proposed approach differs from AC-GAN because it requires no labels and uses an unconditional generator whose images are subsequently rotated.This connects self-supervision to discriminator representation learning rather than class conditioning.
- Self-supervised learning: Self-supervised learning learns high-level semantic representations by solving surrogate tasks.Image rotation prediction is presented as a simple task producing useful representations for downstream classification.
6. Conclusions and Future Work
The paper proposes combining adversarial and self-supervised learning to counter discriminator forgetting, matches equivalent conditional GANs without labels, and reaches FID 23.4 on unconditional ImageNet generation.
- Conclusions: The self-supervised GAN combines adversarial and self-supervised learning to counter discriminator forgetting.The model is described as a deep generative model built from both learning objectives.
- Conclusions: The model can match equivalent conditional GANs on image synthesis without access to labeled data.This result is stated for the self-modulated variant of the self-supervised GAN.
- Conclusions: FID 23.4 is attained on unconditional ImageNet generation after scaling the model.The paper characterizes unconditional ImageNet generation as extremely challenging.
- Future work: Future work includes stronger self-supervised discriminator architectures, semi-supervised fine-tuning, and additional image-synthesis techniques.Suggested techniques include self-attention, orthogonal normalization, regularization, and sampling truncation.
- Future work: The authors position collaborative self-supervision with adversarial training as a direction toward high-quality fully unsupervised generative modeling.This is stated as the broader aspiration of the approach.
A. FID Metric Details
FID is computed from Inception V1 pool_3 embeddings by fitting multivariate Gaussians to real and generated samples.
- Embedding extraction: FID uses image embeddings extracted from the Inception V1 pool_3 layer.The embeddings come from the TensorFlow library implementation.
- Distribution fitting: The metric fits multivariate Gaussians separately to real test-set samples and fake samples.These fitted distributions are used to compute the FID score.
- Sampling protocol: 3000 samples are used for CELEBA-HQ and 10000 samples for the other datasets.The sample counts apply to the real and fake samples used for the metric computation.
B. SS-GAN Hyper-parameters
The appendix studies the self-supervision weight α while fixing β = 1; its preferred value differs between CIFAR10 and ImageNet, with α = 0.2 used in the main experiments.
- Hyper-parameter study: Fixing β = 1, the experiments find optimal α values of 1 for CIFAR10 and 0.2 for ImageNet.The passage describes these values as optimal among those compared.
- Hyper-parameter study: An inappropriate α can bias the generator’s convergence point, whereas a reasonable value helps generator training through the self-supervision task.The main experiments use α = 0.2 for all datasets.
- Hyper-parameter study: Figure 7 evaluates performance under different α values.The figure caption identifies α as the varied hyper-parameter.
C.1. Implementation Details
Linear evaluation models use standardized optimization and augmentation settings, with training conducted on a single NVIDIA Tesla P100 GPU.
- Optimization: Linear evaluation models are trained with batch size 128 and a learning rate of 0.1 × batch_size for 50 epochs.The learning rate is decayed by a factor of 10 after epochs 30 and 40.
- Data augmentation: Images are resized so their smaller dimension is 146, then cropped to 128 × 128 using random crops for training and central crops for testing.
- Hardware: The model is trained on a single NVIDIA Tesla P100 GPU.
C.2. Additional Results
Additional evaluations compare representation quality across datasets, training durations, GAN variants, and FID scores. SS-GAN is consistently strong on CIFAR10 and becomes stable and competitive on deeper IMAGENET representations, while unconditional GANs can collapse.
- Variance and collapse: Unconditional GAN representation quality has large variance on IMAGENET because the unconditional GAN collapses in some cases.Table 7 reports top-1 accuracy with standard deviations for IMAGENET.
- Representation quality: SS-GAN consistently outperforms the other models across all four representation blocks on CIFAR10.Figure 8 reports representation quality across the four CIFAR10 blocks.
- Representation quality: On IMAGENET, methods perform similarly before 500k steps in block0, which contains low-level features.Deeper blocks show larger differences between methods.
- Representation quality: Conditional GAN and SS-GAN representations improve from block0 toward block6, whereas unconditional GAN quality worsens at the last layer and rotation-only quality decreases with training.The conditional GAN benefits from supervised labels near the classification head.
- Representation quality: Combining self-supervised and adversarial losses makes SS-GAN representation quality stable and better than the other models.
- FID correlation: On CIFAR10, representation quality and FID are correlated across methods, but representation quality varies by up to 4% at the same FID score.Figure 10 covers training steps from 10k to 100k.
- FID correlation: On IMAGENET, representation quality and FID are not correlated across blocks 0–4; from block4, SS-GAN representation improves as FID improves.Figure 11 covers training steps from 100k to 1M, with collapsed models having FID scores above 100 removed.