Source-linked AI summary
Transferring GANs: generating images from limited data
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, Bogdan Raducanu
TL;DR
The paper addresses the lack of study of pretrained-network transfer for GAN-based image generation. It evaluates domain adaptation across target-data sizes, source–target distances, and conditional-GAN initialization, finding benefits from pretrained models, particularly with limited data, while identifying density and model-selection challenges.
Problem
Pretrained-network fine-tuning is widely used for discriminative models, but its application to generative deep networks had not been studied.
Method
The paper evaluates GAN domain adaptation across target-domain sizes, source–target relationships, transfer configurations, and unconditional-to-conditional initialization methods.
Results
Pretrained GANs accelerate convergence and improve generated-image quality, with benefits also observed for conditional GANs initialized from unconditional models.
Takeaways & Limitations
Transfer helps GANs and conditional GANs with less training data, while densely sampled source domains may be more useful than diverse datasets.
Takeaways & Limitations
FID between source-generated and target-real data is only a rough suitability indicator and should be combined with source-model quality and other factors.
Abstract
from arXiv · showhide
Transferring the knowledge of pretrained networks to new domains by means of finetuning is a widely used practice for applications based on discriminative models. To the best of our knowledge this practice has not been studied within the context of generative deep networks. Therefore, we study domain adaptation applied to image generation with generative adversarial networks. We evaluate several aspects of domain adaptation, including the impact of target domain size, the relative distance between source and target domain, and the initialization of conditional GANs. Our results show that using knowledge from pretrained networks can shorten the convergence time and can significantly improve the quality of the generated images, especially when the target data is limited. We show that these conclusions can also be drawn for conditional GANs even when the pretrained model was trained without conditioning. Our results also suggest that density may be more important than diversity and a dataset with one or few densely sampled classes may be a better source model than more diverse datasets such as ImageNet or Places.
1 Introduction
The paper examines whether pretrained GANs can transfer knowledge to new image-generation domains, especially when target data is limited. It evaluates transfer configurations, source–target domain relationships, and unconditional-to-conditional GAN initialization.
- 1 Introduction: GANs are typically trained from scratch despite having millions of parameters, motivating pretrained initialization for domains with few training images.The DC-GAN architecture cited requires 36M parameters to generate a 64x64 image.
- 1 Introduction: The paper evaluates whether pretrained networks accelerate GAN learning and provide useful prior knowledge when target data is limited.
- 1 Introduction: The study analyzes how the relationship between source and target domains affects transfer results and how to choose a suitable pretrained model.
- 1 Introduction: It also evaluates transferring knowledge from unconditional GANs to conditional GANs using two commonly used conditioning methods.
2 Related Work
Related work covers transfer learning for discriminative vision models, GAN training and domain-transfer challenges, and conditional GAN architectures. The paper distinguishes its focus on transferring knowledge for image generation rather than discriminative features.
- Transfer learning/domain transfer: Fine-tuning pretrained networks is a standard approach for transferring knowledge to computer-vision tasks with limited data or slow convergence.
- Transfer learning/domain transfer: Prior domain-transfer methods learn domain-invariant or task-related features for unsupervised, sparsely labeled, or metric-learning settings.
- GAN: In contrast to earlier transfer methods, this paper studies transferring knowledge for image generation rather than transferring discriminative features.
- GAN: GAN research addresses training challenges including convergence, instability, mode collapse, unsuitable distributions, and Lipschitz constraints.
- cGAN: Conditional GANs use attributes such as class labels, text, or images as priors, with conditioning applied through inputs, batch normalization, auxiliary decoding, or latent-variable reconstruction.
3 Generative Adversarial Networks
This section defines GANs, their adversarial objectives and variants, and the metrics used to evaluate generated images. FID is the primary metric, while IW is reported only in selected experiments because it is more computationally costly.
- Generative Adversarial Networks: A GAN trains a generator to produce samples resembling real data while a discriminator distinguishes real samples from generated ones.
- Generative Adversarial Networks: The generator maps noise z sampled from p_z to generated samples G(z), and both networks are trained through an adversarial objective.
- Loss functions: WGAN-GP uses a Wasserstein-based loss formulation as an alternative GAN training objective.
- Evaluation Metrics: FID is the primary metric because it is efficient to compute and correlates well with human perception, although GAN evaluation has no universally agreed reference metric.
- Evaluation Metrics: FID compares real and generated datasets in an Inception embedding space by measuring the Fréchet distance between fitted multivariate normal distributions.
- Evaluation Metrics: IW estimates the Wasserstein distance with an independent critic and is reported selectively because each measurement requires training an additional network.
- Evaluation Metrics: Table 1 reports FID and IW for transfer configurations using ImageNet as source data and LSUN Bedrooms as the 100K-image target.
4 Transferring GAN representations
Transferring pretrained GANs accelerates adaptation and improves generated-image quality, especially with limited target data. The experiments also show that discriminator transfer, source-domain selection, and source-model density materially affect outcomes.
- Experimental setup: The WGAN-GP experiments use 64×64 images, pretrained source models, and fine-tuning on target domains with standard optimization settings.Source models use 50K iterations except CelebA; fine-tuning uses batch size 64 and learning rate 1e-4, or 1e-5 for 1K target samples.
- Generator/discriminator transfer configuration: Transferring both generator and discriminator produces the best results and more stable training, while discriminator transfer is more critical than generator transfer.Generator transfer alone can harm performance when the discriminator remains randomly initialized.
- Generator/discriminator transfer configuration: Pretrained GANs reach comparable FID and IW scores in significantly fewer iterations than training from scratch.Training from scratch can reduce the gap with prolonged optimization, but transfer provides good-quality images earlier.
- Size of the target dataset: With scarce target data, adapted GANs require roughly two to five times fewer images than scratch-trained GANs to obtain similar scores.The advantage becomes more significant as the target dataset shrinks, although a substantial gap remains relative to the data representativeness lower bound.
- Source and target domains: Source-target relatedness does not reliably identify the best source model: dense Bedrooms data can outperform more diverse Places data, while FID and IW sometimes disagree.For Kitchens, FID favors Bedrooms over Places, whereas IW slightly prefers Places; Cityscapes performs similarly across source models, with high scores.
- Selecting the pre-trained model: Distances between source-generated and target-real data roughly correlate with final transfer performance, suggesting density may matter more than diversity for transferability.Source-generated-to-target FID accounts for source-model quality, unlike source-real-to-target FID alone, but it remains a rough suitability indicator that should be combined with other factors.
5 Transferring to conditional GANs
The paper transfers unconditional GAN knowledge to conditional GANs using AC-GAN, testing conditioning designs and pretrained versus random initialization. Pretraining accelerates convergence and improves generated-image quality, especially with limited target data.
- Experimental setup: The study transfers pretrained unconditional GAN weights to AC-GANs targeting ten LSUN classes, comparing pretrained and randomly initialized conditional models.Performance is evaluated using class-wise and class-agnostic FID, with Places as the source domain and LSUN as the target.
- AC-GAN formulation: AC-GAN combines the conditional GAN loss with an auxiliary classifier loss that predicts the correct class from generated or real images.The generator and discriminator losses weight the auxiliary classification terms with αG and αD.
- Conditioning variants: The two conditioning variants perform similarly: Cond Concat appends class labels to noise, whereas Cond BNorm uses class-specific batch-normalization parameters.For Cond BNorm, unconditional batch-normalization parameters are copied to all classes before training.
- Results: Pretrained initialization converges faster and retains better overall performance than training from scratch, with the advantage increasing as images per class decrease.The comparison includes 100, 1K, and 10K images per class and evaluates models at multiple training durations.
- Results: Pretrained models achieve higher classifier accuracy, lower FID, and greater human preference for realism, including a 67% preference at 1K images per class.The human comparison shows a smaller difference when using 10K images per class.
6 Conclusions
Pretrained GANs transfer useful generative features, improving image generation with less data, but transferability depends on discriminator initialization, data density, and current GAN limitations.
- Pretrained GANs produce lower FID scores and more recognizable images with less training data.
- Transferring the discriminator is more critical than transferring the generator, although transferring both networks works best.
- Generation appears to require higher image density per class than discrimination, making LSUN a better source choice than ImageNet or Places in these experiments.
- Poor transferability may partly reflect limitations of current GAN techniques, so improved methods could improve transferability.
- Open problems include limited-data GAN evaluation, selecting suitable pretrained models, and designing better pretrained GANs.
A Distances between source and target data
The appendix measures distances between source and target real-image datasets and relates these distances to transfer performance.
- The source-target FID can estimate which pretrained GAN may be suitable for adapting to a target dataset.
- In most cases, the lowest distance in Table A1 also corresponds to the lowest value in Table 1.
- Table A1 reports distances between source real data and target real data.
B Model capacity
A capacity experiment tests whether reducing source-network size affects transfer to LSUN Bedrooms.
- Model capacity: The experiment halves the number of filters in each layer of an ImageNet-trained source GAN.
- Model capacity: The reduced-capacity source GAN is fine-tuned on 10K LSUN Bedrooms images.
- Model capacity: Results are presented in Fig. 6 to assess the importance of network capacity for transferring GAN features.
C Images sampled from the models
The appendix provides visual samples from source models before and after 5K iterations of fine-tuning on four target domains.
- Flowers: Examples compare source-model samples with samples after fine-tuning on Flowers for 5K iterations.
- Kitchens: Examples compare source-model samples with samples after fine-tuning on Kitchens for 5K iterations.
- LFW: Examples compare source-model samples with samples after fine-tuning on LFW for 5K iterations.
- Cityscapes: Examples compare source-model samples with samples after fine-tuning on Cityscapes for 5K iterations.
- Model capacity: Fig. 6 presents the separate model-capacity experiment.
- Each comparison orders models from top to bottom as from scratch, ImageNet, Places, LSUN bedrooms, and CelebA.