Ensembling Off-the-shelf Models for GAN Training

Motivation

• The paper addresses whether pretrained visual representations can improve GAN training and which models and usage strategies are most effective.This question arises because GAN discriminators are typically trained from scratch despite the availability of powerful pretrained vision models.Source: §1 · p. 1

Approach

• Freezing pretrained networks and learning lightweight classifier heads alongside the original discriminator stabilizes training, while multiple feature spaces provide complementary supervision.Naively using strong pretrained networks as discriminators can overfit and overwhelm the generator, especially with limited data.Source: §1 · p. 1
• Linear separability between real and fake samples in pretrained feature space drives automatic model selection and progressive addition of pretrained networks.The procedure selects models from a bank according to their ability to distinguish real and fake distributions, then adds supervision progressively.Source: §1 · pp. 1–2

Results

• 2-3× FID improvement is reported on FFHQ and LSUN with 1k training samples, while LSUN Cat reaches the FID of StyleGAN2 trained on 1.6M images using only 10k samples.These results cover the limited-data setting and are accompanied by differentiable augmentation and label smoothing for stability and reduced overfitting.Source: §1 · p. 2
• Full-scale FID improves from 6.86 to 3.98 on LSUN Cat, from 4.28 to 1.72 on LSUN Church, and from 4.09 to 2.11 on LSUN Horse.The reported improvements span multiple LSUN categories in the large-scale setting.Source: §1 · p. 2

Use of pretrained models in image synthesis

• Prior work uses pretrained features as perceptual losses for conditional synthesis and combines perceptual and adversarial losses in image-to-image translation.Related approaches include perceptual losses for super-resolution, image translation, and style transfer, as well as perceptual discriminators.Source: §2 · p. 2
• This method differs by targeting unconditional GAN training, ensembling diverse feature representations, and automatically selecting models for the domain.The paper contrasts its approach with methods using a single VGG model or perceptual discriminators for other applications.Source: §2 · p. 2

Broader connections

• Other studies use pretrained models for clustering, nearest-neighbor or feature-space constraints, generator conditioning, image editing, and transfer learning, whereas this method equips the discriminator without changing the generator backbone.It transfers learned feature representations from diverse sources rather than transferring generator or discriminator weights.Source: §2 · p. 2

3.3. Training Algorithm

• On LSUN CAT, the method achieves similar FID to StyleGAN2 trained on the full dataset while using only 0.7% of the dataset.Source: Figure 2 · p. 3

3. Method

• A diverse ensemble of pretrained feature representations reduces discriminator overfitting and can better capture visual concepts ranging from edges and textures to objects and parts.The feature extractors remain frozen while lightweight classifier heads are trained, improving validation generalization.Source: §3 · p. 3, §3.1 · p. 3

3.1. Formulation

• The formulation combines the standard discriminator with shallow heads over frozen pretrained extractors, sending their adversarial gradients to the generator.Each vision-aided discriminator measures the real–fake gap in its pretrained feature space.Source: §3.1 · p. 3

3.2. Model Selection

• Linear probing selects models whose pretrained features most accurately distinguish real from generated images, because higher probe accuracy generally correlates with better GAN FID.A logistic head is evaluated on a validation split, and the model with the lowest validation error is selected.Source: §3.2 · p. 4, Figure 4 · p. 4
• K-progressive selection adds the best unused pretrained model after fixed training intervals, reducing computational cost relative to selecting K models at the start.The progressive strategy can select complementary models, often combining supervised and self-supervised representations.Source: §3.2 · p. 4

3.3. Training Algorithm

• Training uses the original discriminator together with selected off-the-shelf models, plus differentiable augmentation and one-sided label smoothing for stabilization.Using only off-the-shelf models as the discriminator can cause divergence, while adding models with low probe accuracy provides minimal benefit.Source: §3.3 · pp. 4–5

4.1. FFHQ and LSUN datasets

• The method improves limited-data GAN quality across FFHQ, LSUN CAT, and LSUN CHURCH, with gains reported from 1k to 10k training samples.The same latent codes generally produce better samples, particularly for FFHQ and LSUN CAT.Source: Figure 5 · p. 5, §4.1 · p. 6
• Higher linear-probe accuracy generally identifies pretrained models that yield better FID, motivating selection of the next discriminator during progressive ensemble training.On FFHQ 1k, ViT (DINO), ViT (CLIP), and Swin-T (MoBY) are selected in that order as probe accuracy changes.Source: §4.1 · p. 6, Figure 6 · p. 6
• FID improves from 6.86 to 3.98 on LSUN CAT, from 4.28 to 1.72 on LSUN CHURCH, and from 4.09 to 2.11 on LSUN HORSE in full-dataset training.On FFHQ, PPL improves from 144.62 to 127.58 while FID remains similar.Source: §4.1 · p. 6, Table 2 · p. 6
• GAN Dissection finds that interpretable generator units increase from 254 to 297 on LSUN CHURCH, suggesting richer semantic representations.The analysis also reports units with improved IoU to semantic categories.Source: §4.1 · p. 6, Figure 7 · p. 6

4.2. AFHQ and METFACES

• On AFHQ and METFACES, the method generally achieves lower FID and higher Recall than existing methods across fine-tuning and transfer settings.The transfer setup fine-tunes a same-resolution FFHQ-trained model using FreezeD.Source: Table 3 · p. 7, §4.2 · p. 8

4.3. Low-shot Generation

• The method significantly improves FID and KID over leading few-shot GAN methods on 100-shot Bridge-of-Sighs and AnimalFace cat and dog datasets.Experiments use 169 AnimalFace cat images, 389 dog images, and 100 Bridge-of-Sighs samples.Source: Table 4 · p. 7, §4.3 · p. 8

4.4. Ablation Study

• Progressive model addition reaches similar FID with lower runtime than fixed multi-model training, while random selection helps but performs worse than linear-probe selection.Without data augmentation, limited-data training collapses early; label smoothing is especially useful in limited-data settings, and DiffAugment is comparable to ADA.Source: Figure 10 · p. 8, §4.4 · pp. 8–9, Table 5 · p. 8
• Additional ablations show that random or trainable extra discriminators and longer standard adversarial training provide only marginal FID benefits compared with the proposed discriminator ensemble.The reported improvement is attributed to combining the original discriminator with vision-aided discriminators.Source: §4.4 · p. 9, Table 6 · p. 9

5. Limitations and Discussion

• Multiple pretrained discriminators improve generated-image quality, especially in limited-data settings, but increase training memory requirements.The authors suggest efficient computer vision models as a possible way to improve accessibility.Source: §5 · p. 9
• Model selection becomes unreliable with roughly 100 samples because linear-probe accuracy has increased variance, motivating few-shot-learning methods for low-shot settings.The authors identify this as a limitation rather than a general failure of the training method.Source: §5 · p. 9
• The paper presents pretrained representation transfer as a step toward using increasingly available supervised and self-supervised vision models for generative modeling.The appendix covers additional qualitative, quantitative, BigGAN, metric, and training-detail analyses.Source: §5 · p. 9, Appendix · p. 14

A. Qualitative Image Analysis

• Across FFHQ, LSUN CAT, and LSUN CHURCH, many qualitative comparisons show improved visual quality over the baseline methods.The comparisons include matched latent codes and varying training sample settings.Source: A. Qualitative Image Analysis · p. 14
• Low-shot nearest-neighbor tests and latent interpolations are used to assess whether models overfit, with smooth interpolation reported for models trained on 100–400 samples.The smooth interpolation is interpreted as probable evidence against overfitting to the few real samples.Source: A. Qualitative Image Analysis · p. 14
• The appendix includes worst-case sample comparisons for AFHQ and METFACES using Gaussian likelihoods in Inception feature space.Samples with minimum likelihood, equivalently maximum Mahalanobis distance, are displayed.Source: A. Qualitative Image Analysis · p. 14

B. Vision-aided BigGAN

• Vision-aided adversarial training outperforms DiffAugment in FID across all evaluated BigGAN settings on CIFAR-10 and CIFAR-100.The comparison varies training dataset size and includes both unconditional and conditional generation.Source: B. Vision-aided BigGAN · p. 14
• The BigGAN experiments use self-modulation for unconditional generation and projection discriminators for conditional vision-aided training.Training runs use the same number of iterations as DiffAugment.Source: B. Vision-aided BigGAN · p. 14
• Linear-probe accuracy on pretrained features correlates with FID in the LSUN CAT 1k model-selection experiment.The figure uses the StyleGAN2-ADA generator during selection and reports better FID for models with higher probe accuracy.Source: Figure 12 · p. 14

• Our method improves recall across LSUN experiments while maintaining similar or better precision, especially in limited-sample settings.Source: C. Evaluation · p. 14
• SwAV-FID evaluations show consistent improvements when measuring FID in a self-supervised ImageNet feature space.Source: C. Evaluation · p. 14
• Using CLIP or DINO as vision-aided discriminators also improves FID despite their differing pretraining setups.Source: C. Evaluation · p. 14

• The model bank includes state-of-the-art feature extractors and task-specific networks paired with small, broadly consistent discriminator heads.Source: D. Training and Hyperparameter details · p. 14
• CLIP, DINO, and MoBY are selected for CIFAR experiments, improving FID in both conditional and unconditional BigGAN training.Source: Table 7 · p. 15
• CLIP and DINO use multi-scale heads over intermediate spatial features and the final classification token, yielding slightly better FID than a token-only head.Source: D. Training and Hyperparameter details · p. 15, Table 8 · p. 15
• Model selection ranks pretrained networks by linear-probe accuracy on real and fake samples, with low variance except in the smallest-data setting.Source: D. Training and Hyperparameter details · p. 15
• Training uses one-sided label smoothing, color, translation, and cutout augmentation, with ADA targets and regularization adjusted by dataset and setting.Source: D. Training and Hyperparameter details · pp. 15–17
• Precision and recall are evaluated alongside FID, with higher recall generally preferred because truncation can recover precision.Source: Table 10 · p. 16, Table 9 · p. 16
• The ensemble contains three vision-aided discriminators added progressively, with memory overhead ranging from roughly 10–25% for one model to about 4.5GB for three selected models.Source: D. Training and Hyperparameter details · p. 17

• The proposed method is aimed specifically at improving GAN image quality in limited-sample settings.Source: E. Societal Impact · p. 17
• It may support novel content creation when users have only a few relevant samples for inspiration.Source: E. Societal Impact · p. 17
• Faster convergence and lower computational cost could broaden access to GAN training, while also enabling fake data and disinformation.Source: E. Societal Impact · p. 17
• Reliable detection of generated fake data is suggested as one possible mitigation for these negative societal impacts.Source: E. Societal Impact · p. 17

F. Change log

• The revised paper adds Figure 14, comparing the method with DiffAugment in paired qualitative samples across LSUN Cat, FFHQ, and LSUN Church.Source: F. Change log · p. 17, Figure 14 · p. 20
• Table 15 adds intermediate results for progressively adding pretrained discriminators on METFACES and AFHQ.Source: F. Change log · p. 17, Table 15 · p. 18
• The revisions add CIFAR BigGAN results, SwAV-FID evaluations, and a nearest-neighbor test for low-shot models.Source: F. Change log · p. 17
• Table 16 documents the off-the-shelf model bank and discriminator-head designs, including multi-scale CLIP and DINO features.Source: Table 16 · p. 18

FFHQ 2k

• Paired samples in the 1k-data ADA setting report average image-quality improvements for the method over StyleGAN2-ADA using matched latent codes.Source: Figure 13 · p. 19
• Human preference rates favor the method by 63.5% on LSUN Cat, 53.8% on FFHQ, and 60.5% on LSUN Church.Source: Figure 15 · p. 21
• Figures 16 and 17 compare DiffAugment, StyleGAN2-ADA, and the method on FFHQ and LSUN Church at 2k and 10k samples.Source: Figure 16 · p. 22, Figure 17 · p. 23

LSUN HORSE

• Full-dataset qualitative samples compare StyleGAN2 with the proposed method using ADA on LSUN HORSE.Source: Figure 18 · p. 24

Qualitative comparisons

• The proposed method shows better average image quality than StyleGAN2-ADA in qualitative comparisons on AFHQ DOG and AFHQ CAT.Source: Figure 19 · p. 25

Low-shot qualitative results

• On 1k-sample settings, the method reduces extreme rotated artifacts on FFHQ and has relatively better worst-case samples on LSUN CHURCH.Source: Figure 21 · p. 27, Figure 22 · p. 28

Low-shot diagnostics

• Nearest-neighbor comparisons indicate that generated images differ from the training set, while smooth latent interpolations suggest little overfitting in several low-shot datasets.Source: Figure 23 · p. 29, Figure 24 · p. 30

Progressive ensemble diagnostics

• As more vision-aided discriminators are added, pretrained-model linear-probe accuracy decreases across datasets and training settings.Source: Figure 25 · p. 31, Figure 26 · p. 32, Figure 27 · p. 33