Source-linked AI summary
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
Sangwoo Mo, Minsu Cho, Jinwoo Shin
TL;DR
GAN transfer learning seeks to reduce GANs’ data and computational demands, but existing approaches can overfit or handle only limited distribution shifts. The paper proposes FreezeD, which freezes lower discriminator layers while fine-tuning upper layers. Across unconditional and conditional GAN experiments, FreezeD outperforms previous techniques, though stability worsens for Oxford Flower.
Problem
GANs often require substantial training data and computational resources, while prior transfer-learning methods can overfit or fail to robustly learn significant distribution shifts.
Method
FreezeD splits the discriminator into a feature extractor and classifier, freezes lower layers, and fine-tunes the upper layers.
Results
FreezeD clearly outperforms most previous methods across unconditional and conditional GANs using various architectures and datasets.
Takeaways & Limitations
FreezeD provides a simple, effective baseline for GAN transfer learning and a reference point for developing more sophisticated methods.
Takeaways & Limitations
FreezeD improves performance and stability for most conditional cases but harms stability for Oxford Flower, where feature distillation is more stable.
Abstract
from arXiv · showhide
Generative adversarial networks (GANs) have shown outstanding performance on a wide range of problems in computer vision, graphics, and machine learning, but often require numerous training data and heavy computational resources. To tackle this issue, several methods introduce a transfer learning technique in GAN training. They, however, are either prone to overfitting or limited to learning small distribution shifts. In this paper, we show that simple fine-tuning of GANs with frozen lower layers of the discriminator performs surprisingly well. This simple baseline, FreezeD, significantly outperforms previous techniques used in both unconditional and conditional GANs. We demonstrate the consistent effect using StyleGAN and SNGAN-projection architectures on several datasets of Animal Face, Anime Face, Oxford Flower, CUB-200-2011, and Caltech-256 datasets. The code and results are available at https://github.com/sangwoomo/FreezeD.
1. Introduction
GAN transfer learning addresses the data and resource demands of GANs, but prior methods can overfit or struggle with substantial distribution shifts. FreezeD instead freezes lower discriminator layers during fine-tuning and performs well across evaluated settings.
- GANs often require large training datasets and heavy computational resources, limiting their applicability in practical scenarios.
- Prior GAN transfer-learning methods are prone to overfitting with limited data or lack robustness to significant distribution shifts.
- FreezeD fine-tunes both GAN components while freezing lower discriminator layers, which learn generic image features, and adapts upper layers for real-versus-fake classification.
- FreezeD outperforms previous techniques across unconditional and conditional settings using StyleGAN and SNGAN-projection on five datasets.The evaluated datasets are Animal Face, Anime Face, Oxford Flower, CUB-200-2011, and Caltech-256.
- 61.46 FID, versus 64.28 for fine-tuning, is reported for FreezeD on the Dog class of Animal Face, a −4.4% change.
2. Methods
The paper positions FreezeD as a simple GAN transfer-learning baseline that freezes lower discriminator layers and fine-tunes upper layers. It contrasts this approach with prior methods and reports favorable performance with lower computational cost than feature distillation.
- Prior methods: Fine-tuning initializes target models from pretrained source weights but often suffers from overfitting, motivating regularization.
- Prior methods: Scale/shift updates normalization layers while fixing other weights, but can perform poorly when source and target distributions differ substantially.
- Prior methods: GLO fine-tunes the generator and latent codes with L1 and perceptual losses, improving stability but tending to produce blurry images.
- Prior methods: MineGAN fixes the generator and trains a miner network to transform latent codes, but may not generalize when source and target supports are disjoint.
- FreezeD: FreezeD freezes lower discriminator layers and fine-tunes only the upper layers as a simple baseline for GAN transfer learning.
- Prior methods: L2-SP regularizes target parameters toward source parameters, but the tested generator, discriminator, and joint variants were unsatisfactory.
- Prior methods: Feature distillation produces results comparable to FreezeD while requiring twice the computation in the reported experiments.
3. Experiments
Experiments evaluate FreezeD across unconditional and conditional GANs, comparing it with fine-tuning and prior transfer-learning methods using quantitative and qualitative results. FreezeD generally improves performance and stability, although stability decreases for Oxford Flower.
- Method comparisons: Feature distillation shows comparable results to FreezeD but requires twice the computation and is more stable in SNGAN-projection experiments.The paper identifies advanced feature-distillation methods as a possible future direction.
- Qualitative results: Using the same latent code after fine-tuning preserves semantics such as background color and hair color in StyleGAN samples.Figure 2 compares original weights with FreezeD models trained on the ‘Cat’ and ‘Dog’ classes.
- Experimental setup: Experiments cover unconditional StyleGAN adaptation to Animal Face and Anime Face and conditional SNGAN-projection adaptation to Oxford Flower, CUB-200-2011, and Caltech-256.Animal Face uses 20 classes and Anime Face uses 10 classes; the conditional datasets contain 102, 200, and 256 classes, respectively.
- Unconditional GAN: FreezeD freezes discriminator layers while fine-tuning the models, improving both best and final FID scores for the unconditional experiments.For Animal Face and Anime Face, the discriminator is frozen through layer 4.
- Method comparisons: Scale/shift and L2-SP harm diversity, GLO produces blurry images, and MineGAN fails to learn the distribution shift.These comparisons are reported for Animal Face experiments involving the ‘Cat’ and ‘Dog’ classes.
- Conditional GAN: FreezeD produces more class-consistent Oxford Flower samples than fine-tuning, while conditional experiments show improved performance and stability in most cases.The qualitative comparison highlights abnormal fine-tuning samples in rows 2 and 8; discriminator freezing uses 3, 2, and 1 layers for Oxford Flower, CUB-200-2011, and Caltech-256, respectively.
4. Conclusion
The paper concludes that FreezeD is a simple and effective baseline for transfer learning in GANs. It reports broad performance improvements and proposes discriminator feature transfer and more advanced methods as future directions.
- Conclusion: FreezeD splits the discriminator into a feature extractor and classifier, then fine-tunes only the classifier.The approach is presented as a baseline for transfer learning of GANs.
- Conclusion: FreezeD clearly outperforms most previous methods across various architectures and datasets.The conclusion frames FreezeD as a baseline rather than a claim to state-of-the-art performance.
- Future directions: The paper suggests applying discriminator feature-extractor transfer to universal generated-image detection and developing more sophisticated methods.Advanced feature distillation is identified as a promising direction.
Supplementary Material:
The paper is titled “Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs.”
- Title: The title names FreezeD as a discriminator-freezing baseline for fine-tuning GANs.The title introduces the paper’s central method and application area.
A. Ablation Study on Freezing Layers
The best number of frozen discriminator layers depends on the GAN architecture and target distribution shift. Layer 4 is best for StyleGAN, while SNGAN-projection favors different freezing depths across datasets.
- SNGAN-projection: SNGAN-projection performs best with layers {3, 2, 1} frozen for Oxford Flower, CUB-200-2011, and Caltech-256, respectively.The preferred depth varies across the three target datasets.
- Interpretation: Larger distribution shifts require less restriction so the model can adapt more freely.The paper identifies Caltech-256 as harder to learn than Oxford Flower because its distribution shift is larger.
- Stability: FreezeD is less stable than fine-tuning on Oxford Flower.The paper notes that feature distillation is more stable while achieving similar best performance in early experiments.
- StyleGAN: Layer 4 performs best for StyleGAN on the Animal Face ‘Cat’ and ‘Dog’ classes.Layer i freezes the first i discriminator layers.
B. Comparison to Feature Distillation
Feature distillation achieves results comparable to FreezeD but requires more computation. The comparison evaluates matching discriminator activations across layers.
- Performance: Feature distillation shows comparable results to FreezeD on StyleGAN for the Animal Face dataset.The comparison covers the ‘Bear,’ ‘Cat,’ and ‘Dog’ classes.
- Efficiency: Feature distillation is twice slower than FreezeD.Feature distillation matches normalized discriminator activations from source and target models.
C. Qualitative Results for Prior Methods
Prior transfer-learning methods differ qualitatively in adaptation quality and diversity. Scale/shift and L2-SP produce reasonable but less diverse samples, while GLO and MineGAN show more severe limitations.
- Sample quality: Scale/shift and L2-SP generate reasonable samples but exhibit less diversity according to FID scores.The comparison is shown for samples generated under the ‘Dog’ class in Animal Face.
- Sample quality: GLO generates blurry images because it lacks adversarial loss and source-discriminator knowledge.
- Distribution shift: MineGAN totally fails to adapt to the target distribution in these experiments.The paper notes that MineGAN assumes the source distribution covers or is close to the target distribution, which does not hold for disjoint supports such as human faces and dog faces.
E. Generated Samples by SNGAN-projection
The generated-sample panels compare fine-tuning and FreezeD on CUB-200-2011 and Caltech-256. The supplied passages identify the dataset and method for each panel but do not state a visual outcome.
- CUB-200-2011: Panels (a) and (b) show CUB-200-2011 samples from fine-tuning and FreezeD, respectively.
- Caltech-256: Panels (c) and (d) show Caltech-256 samples from fine-tuning and FreezeD, respectively.