PaperlayerView exact revision on arXiv

Source-linked AI summary

Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs

Sangwoo Mo, Minsu Cho, Jinwoo Shin

arXiv:2002.10964v2cs.CVcs.LGstat.ML

TL;DR

GAN transfer-learning methods can overfit limited data or struggle with significant distribution shifts. FreezeD fine-tunes GANs while freezing the discriminator’s lower layers, and outperforms previous techniques across the reported experiment settings.

  • Problem

    GAN transfer-learning methods can overfit with limited data or fail to robustly learn significant distribution shifts.

    Source: §1 · p. 1

  • Method

    FreezeD fine-tunes GANs while freezing the discriminator’s lower layers and fine-tuning its upper layers.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    Across all experiment settings, FreezeD outperformed previous techniques; on Animal Face’s Dog class, FID improved from 64.28 with fine-tuning to 61.46 (-4.4%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    FreezeD establishes a simple, effective baseline for GAN transfer learning and encourages development of techniques that outperform it.

    Source: §2 · p. 2, §4 · p. 4

    FreezeD harms stability on Oxford Flower, where feature distillation shows more stable results in these experiments.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Generative adversarial networks (GANs) have shown outstanding performance on a wide range of problems in computer vision, graphics, and machine learning, but often require numerous training data and heavy computational resources. To tackle this issue, several methods introduce a transfer learning technique in GAN training. They, however, are either prone to overfitting or limited to learning small distribution shifts. In this paper, we show that simple fine-tuning of GANs with frozen lower layers of the discriminator performs surprisingly well. This simple baseline, FreezeD, significantly outperforms previous techniques used in both unconditional and conditional GANs. We demonstrate the consistent effect using StyleGAN and SNGAN-projection architectures on several datasets of Animal Face, Anime Face, Oxford Flower, CUB-200-2011, and Caltech-256 datasets. The code and results are available at https://github.com/sangwoomo/FreezeD.

1. Introduction

FreezeD is a simple GAN transfer-learning baseline that fine-tunes both networks while freezing the discriminator’s lower layers. It outperforms previous techniques across the reported experiment settings.

1. Introduction

  • GANs have succeeded across computer vision, graphics, and machine learning applications, but state-of-the-art models often need substantial training data and compute, limiting practical use.The introduction lists image generation [5] [20] [21], image-to-image translation [28] [33] [8], and video-to-video synthesis [43] [3] [6] as examples.Source: §1 · p. 1
  • FID improved from 64.28 with fine-tuning to 61.46 with FreezeD (-4.4%) on Animal Face’s ‘Dog’ class, and FreezeD outperformed prior techniques in all settings.The figure compares FID trends for fine-tuning and FreezeD on this class.Source: Figure 1 · p. 1, §1 · p. 2
  • Existing GAN transfer methods can overfit with limited data or fail to learn significant distribution shifts [32] [44].The introduction frames transfer learning as promising when training data and resources are limited.Source: §1 · p. 1
  • FreezeD fine-tunes both the generator and discriminator while freezing the discriminator’s lower layers, which are treated as generic image-feature extractors.The discriminator’s upper layers classify images as real or fake; the feature-extractor/classifier distinction is borrowed from classifier fine-tuning.Source: §1 · p. 1
  • Experiments cover unconditional StyleGAN fine-tuning from FFHQ to Animal Face and Anime Face, and conditional SNGAN-projection fine-tuning from ImageNet to Oxford Flower, CUB-200-2011, and Caltech-256.The GANs’ pretraining and datasets are identified in the paper as StyleGAN [20], FFHQ [20], Animal Face, Anime Face, SNGAN-projection [27], ImageNet, Oxford Flower, CUB-200-2011, and Caltech-256.Source: §1 · p. 2

2. Methods

Methods introduces FreezeD as a simple GAN transfer-learning baseline and examines it alongside prior adaptation strategies.

Prior transfer-learning methods

  • Fine-tuning initializes target weights from source weights and reportedly performs best when applied to both networks, but risks overfitting; scale/shift [32] updates normalization alone and may underperform under large shifts.Scale/shift fixes all other weights, limiting its ability to adapt when the source and target distributions differ substantially.Source: §2 · p. 2
  • GLO [32] [4] fine-tunes the generator using L1 and perceptual loss [19] while jointly optimizing latent codes, improving stability but tending to produce blurry images without adversarial loss.Each latent code corresponds to one real sample, and interpolation between codes enables sample generalization.Source: §2 · p. 2
  • MineGAN [44] freezes the generator and trains a miner to transform latent codes; it may work when source and target supports overlap but fail when they are disjoint.Source: §2 · p. 2

FreezeD

  • FreezeD freezes the discriminator’s lower layers and fine-tunes its upper layers, and is presented as a simple baseline that outperforms prior methods.The paper frames FreezeD as an effective baseline, not a claim to state-of-the-art, and aims to encourage methods that improve on it.Source: §2 · p. 2

Prior transfer-learning methods

  • L2-SP [23] regularizes target parameters to stay near source parameters; tests on the generator, discriminator, or both were unsatisfactory, though layer-specific weights might improve it.The authors relate layer freezing to assigning infinite L2-SP weight to selected layers and zero weight to the others.Source: §2 · p. 2
  • Feature distillation [16] [35] achieves results comparable to FreezeD at twice the computation cost; more advanced distillation methods [1] [18] [34] are suggested as future work.Source: §2 · p. 2

3. Experiments

Experiments evaluate FreezeD on unconditional StyleGAN and conditional SNGAN-projection tasks, finding improvements over vanilla fine-tuning with some stability caveats.

3.2. Conditional GAN

  • Feature distillation shows more stable results than FreezeD in SNGAN-projection experiments, with similar best results.The paper identifies this stability comparison as an issue for future investigation.Source: §3 · p. 2, §3.2 · p. 4

3.1. Unconditional GAN

  • On Animal Face, the same latent code retains its semantics after fine-tuning, while FreezeD produces more class-consistent conditional samples than vanilla fine-tuning.The unconditional examples preserve features such as background and hair color; conditional examples include fewer abnormal samples in rows 2 and 8.Source: Figure 2 · p. 3, §3.1 · p. 3, Figure 3 · p. 4

3. Experiments

  • FreezeD improves best and final FID and stability on unconditional datasets and most conditional cases, but harms stability on Oxford Flower.The unconditional results cover Animal Face and Anime Face; conditional results cover Oxford Flower, CUB-200-2011, and Caltech-256.Source: Table 1 · p. 3, Table 2 · p. 3, §3.1 · p. 3

3.1. Unconditional GAN

  • Unconditional experiments fine-tune pretrained 256×256 StyleGAN for 50,000 iterations on all 20 Animal Face classes and 10 Anime Face classes.Each class has around 100 samples; training succeeds without progressive training by utilizing the source models.Source: §3.1 · p. 3
  • Scale/shift and L2-SP restrict diversity, GLO produces blurry images, and MineGAN fails to learn the distribution shift.Source: §3.1 · p. 4

3.2. Conditional GAN

  • Conditional experiments fine-tune pretrained 128×128 SNGAN-projection for 20,000 iterations on Oxford Flower, CUB-200-2011, and Caltech-256.The datasets contain 102, 200, and 256 classes, respectively, with 50–100 samples per class; discriminator freezing extends to fewer layers as distribution shift grows.Source: §3.2 · p. 4

4. Conclusion

FreezeD is a simple GAN transfer-learning baseline that fine-tunes only the discriminator’s classifier, and it outperforms most previous methods across architectures and datasets. The authors identify generated-image detection and more sophisticated feature distillation as directions for future work.

4. Conclusion

  • FreezeD splits the discriminator into a feature extractor and classifier, then fine-tunes only the classifier for GAN transfer learning.Source: §4 · p. 4
  • The baseline clearly outperforms most previous methods across various architectures and datasets.Source: §4 · p. 4
  • The authors suggest applying discriminator feature-extractor transferability to universal generated-image detection [41].Source: §4 · p. 4
  • They hypothesize that advanced feature distillation [16] [35] could support a more sophisticated method that surpasses FreezeD.Source: §4 · p. 4

Supplementary Material: · A. Ablation Study on Freezing Layers

The ablation finds different best discriminator-freezing depths across architectures and datasets, while also reporting stability differences on Oxford Flower.

A. Ablation Study on Freezing Layers

B. Comparison to Feature Distillation

Feature distillation achieves results comparable to FreezeD on Animal Face classes, but takes twice as long; the paper therefore selects FreezeD as its baseline. The comparison matches source and target discriminator activations using a layer-normalized L2 regularizer.

B. Comparison to Feature Distillation

  • The distillation method matches linearized source and target discriminator activations using feature-dimension-normalized L2 loss with regularizer weight 1 at every layer.The feature activations vary in size by layer; the regularizer weight is fixed regardless of layer.Source: B. Comparison to Feature Distillation · p. 8
  • Feature distillation is comparable to FreezeD but twice slower on StyleGAN ‘Bear’, ‘Cat’, and ‘Dog’ classes, so FreezeD is chosen as the baseline.Table 7 compares FreezeD with feature distillation (FD) on the Animal Face dataset.Source: Table 7 · p. 8, B. Comparison to Feature Distillation · p. 8

C. Qualitative Results for Prior Methods

Prior methods show distinct shortcomings on Dog samples: Scale/shift and L2-SP produce reasonable but less diverse samples, GLO produces blurry images, and MineGAN fails to adapt to the target distribution.

C. Qualitative Results for Prior Methods

D. Generated Samples by StyleGAN · E. Generated Samples by SNGAN-projection

The supplied passages label SNGAN-projection generated-sample panels for CUB-200-2011 and Caltech-256 under fine-tuning and FreezeD; they provide no description of the samples or evidence for subsection D.

E. Generated Samples by SNGAN-projection

Related papers

Loading 2002.10964v2…