Source-linked AI summary
MineGAN: effective knowledge transfer from GANs to target domains with few images
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, Joost van de Weijer
TL;DR
MineGAN targets the limited attention and high data and training costs of knowledge transfer for generative models. It mines target-relevant regions from one or more pretrained GANs before finetuning, and experiments report effective transfer with few target images, outperforming existing methods while supporting multiple sources.
Problem
Knowledge transfer for generative models has received less attention despite the substantial data and training effort required to train GANs and the need to adapt them using few target images.
Method
MineGAN uses a miner network to steer one or more pretrained GANs toward latent regions whose generated images resemble the target domain, followed by generator finetuning.
Results
MineGAN effectively transfers knowledge to domains with few target images, outperforms existing methods across several settings, and transfers knowledge from multiple pretrained GANs.
Takeaways & Limitations
Mining provides a targeted transfer mechanism for adapting pretrained GANs to single or multiple target domains without access to the large pretraining datasets.
Takeaways & Limitations
Off-manifold target domains are more challenging because their supports may be disjoint from the original data manifold, and BSA produced blur artifacts in one conditional setting.
Abstract
from arXiv · showhide
One of the attractive characteristics of deep neural networks is their ability to transfer knowledge obtained in one domain to other related domains. As a result, high-quality networks can be trained in domains with relatively little training data. This property has been extensively studied for discriminative networks but has received significantly less attention for generative models. Given the often enormous effort required to train GANs, both computationally as well as in the dataset collection, the re-use of pretrained GANs is a desirable objective. We propose a novel knowledge transfer method for generative models based on mining the knowledge that is most beneficial to a specific target domain, either from a single or multiple pretrained GANs. This is done using a miner network that identifies which part of the generative distribution of each pretrained GAN outputs samples closest to the target domain. Mining effectively steers GAN sampling towards suitable regions of the latent space, which facilitates the posterior finetuning and avoids pathologies of other methods such as mode collapse and lack of flexibility. We perform experiments on several complex datasets using various GAN architectures (BigGAN, Progressive GAN) and show that the proposed method, called MineGAN, effectively transfers knowledge to domains with few target images, outperforming existing methods. In addition, MineGAN can successfully transfer knowledge from multiple pretrained GANs. Our code is available at: https://github.com/yaxingwang/MineGAN.
1. Introduction
MineGAN addresses knowledge transfer for generative models by mining useful regions of pretrained GANs for targeted generation from few images. It supports transfer from single or multiple GANs and reports stronger performance than existing competitors across several settings.
- High-quality GANs require substantial training data and time, motivating reuse of pretrained models for new target distributions.Progressive GANs, for example, are trained on 30K images and may require a month on one NVIDIA Tesla V100.
- MineGAN introduces a miner network that steers a pretrained GAN’s latent sampling toward a target distribution defined by few images.The miner transforms a multivariate normal distribution into a distribution on the pretrained GAN’s input space.
- Mining narrows sampling toward target-relevant latent regions, easing subsequent finetuning with a cleaner, lower-variance training signal.The miner has considerably fewer parameters than the pretrained GAN and is therefore less prone to overfitting.
- The method transfers knowledge from multiple pretrained GANs, aggregating information from several sources to generate samples resembling the target domain.
- MineGAN outperforms existing competitors across unconditional, conditional, and multiple-GAN transfer settings.
2. Related work
Prior GAN transfer methods adapt pretrained generators to scarce target data but can suffer from mode collapse, overfitting, or limited adaptation flexibility. MineGAN instead adapts the input distribution and extends transfer to multiple GANs.
- Knowledge transfer has been widely studied for discriminative models but remains less explored for generative models, particularly with few target images.
- Finetuning pretrained GANs can improve performance on limited-sample target domains but suffers from mode collapse and overfitting.
- Batch-normalization-only adaptation supports style changes but is not expected to handle target domains requiring shape changes.
- That alternative also requires truncated input noise during inference to generate realistic samples, whereas MineGAN automatically adapts the input distribution.
- MineGAN is presented as the first method to transfer knowledge from multiple GANs to a single target domain.
3. Mining operations on GANs
MineGAN adapts one or more pretrained GANs to a target distribution with few images by mining suitable latent regions before finetuning. It supports single- and multi-GAN transfer, including conditional generation, and distinguishes on-manifold from more difficult off-manifold targets.
- Setup: MineGAN assumes access to one or more pretrained GANs and uses only few target-domain images to train a new generator for that domain.The method is presented for single-GAN mining first and generalized to arbitrary families of pretrained generators.
- Single-GAN mining: A miner transforms noise into a targeted prior that selects pretrained-generator regions whose outputs better approximate the target distribution.The pretrained generator remains fixed during mining, while gradients update the miner through generated target-domain samples.
- Single-GAN mining: MineGAN uses two stages: first mining with the generator fixed, then joint finetuning of the miner, generator, and discriminator.The first stage is called MineGAN (w/o FT), whereas MineGAN denotes the full method including finetuning.
- Target settings: On-manifold targets overlap the source distribution, whereas off-manifold targets have negligible overlap and require finding samples beyond the original distribution’s support.The paper identifies off-manifold generation as more challenging; cat faces transferred from a human-face source illustrate this case.
- Multiple GANs: For multiple GANs, separate miners identify useful subregions and a selector learns generator sampling frequencies from critic scores.Supersamples contain one generated sample per pretrained generator; the selector can prioritize a source or mix sources according to the target distribution.
- Conditional and multi-GAN extensions: In conditional BigGAN mining, an additional miner estimates the class embedding, while experiments report lower errors for conditional than unconditional models and higher scores off-manifold than on-manifold.The conditional model selects the target class more often, and off-manifold scores remain reasonably low despite the greater difficulty.
4. Experiments
Experiments evaluate MineGAN across unconditional and conditional GANs, on- and off-manifold targets, and transfers from single or multiple pretrained models. Across these settings, MineGAN generally improves image quality and supports knowledge transfer with few target images, including cases where target labels or multiple sources are involved.
- Evaluation setup: The evaluation uses FID, KMMD, and MV across Progressive GAN, SNGAN, and BigGAN experiments, with target domains containing limited data.FID measures quality and diversity but can be unstable on small datasets; KMMD and MV provide additional quality and diversity measures.
- Single pretrained model: On MNIST, mining alone steers a pretrained generator toward unseen target digits by merging patterns from other source digits.The experiment uses 1,000 target images and omits finetuning.
- Single pretrained model: MineGAN outperforms baselines on CelebA→FFHQ women and children, while mining alone nearly matches TransferGAN on the off-manifold children task.On-manifold performance improves further after finetuning, and generated images avoid the overfitting seen with scratch training and sometimes TransferGAN.
- Multiple pretrained models: MineGAN transfers knowledge from multiple pretrained GANs to red vehicles, Tower, and Bedroom targets, outperforming TransferGAN and identifying relevant source models.For red vehicles, the target distributions vary across car-to-bus ratios, while the multi-source experiments use 200 target images.
- Conditional GANs: In conditional BigGAN experiments, MineGAN obtains the best FID and KMMD scores for both on-manifold and off-manifold targets without using target-label information.MineGAN also produces realistic off-manifold results, while TransferGAN degrades when the target label is absent from the pretrained GAN.
- Conditional GANs: Unlike DGN-AM and PPGN, MineGAN uses feedforward inference rather than successive backpropagation updates until convergence.The comparison reports inference times using 200 iterations for DGN-AM and PPGN.
5. Conclusions
MineGAN transfers knowledge for generative models by mining target-relevant regions of learned GAN manifolds before fine-tuning. It supports single and multiple pretrained GANs and was effective across architectures and datasets.
- MineGAN identifies regions of pretrained GAN manifolds closer to a target domain, enabling more effective and efficient fine-tuning with few target images.
- The method applies to both single and multiple pretrained GANs.
- Experiments with BigGAN, Progressive GAN, and SNGAN across multiple datasets demonstrated MineGAN's effectiveness.
A. Architecture and training details
The implementation uses miners alongside generators and discriminators, with architectures and optimization settings adapted to different datasets and GAN configurations. Conditional models can be handled by treating each conditioning value as an independently pretrained generator.
- MNIST: For MNIST, the miner uses two fully connected layers matching the latent-space dimensionality, with |z| = 16 for visual results and |z| = 128 for improved quantitative results.
- MNIST: Conditional MNIST uses a one-hot label vector for conditioning, differing from BigGAN's conditioning scheme.
- Conditional models: A conditional generator G(c, z) is extended by treating each G(i, z) as an independently pretrained generator, with a selector predicting the conditioning label.
- Progressive GAN datasets: For CelebA, FFHQ Children, and LSUN, the Progressive GAN-based miner uses four fully connected layers with ReLU activations and pixel normalization except at the last layer.
- SNGAN datasets: For FFHQ Face and Anime Face, the SNGAN-based miner uses three fully connected layers with sizes (8-32-64-128).
- BigGAN datasets: For ImageNet and Places365, the BigGAN miner contains separate M_z and M_c subnetworks, each composed of fully connected layers.
B. Evaluation metric details
Evaluation measures distance between generated and real images using FID and KMMD, with sample counts adjusted to the available target-set size. KMMD is included because FID can be unstable on small datasets.
- FID is computed between 10,000 generated and 10,000 real images when possible.
- When target datasets contain fewer than 10,000 images, the number of generated images is capped at the target-set size for FID computation.
- KMMD is also used to evaluate generated-real image distance because FID suffers from instability on small datasets.
C. Ablation study
The ablation study examines selection aggregation and miner depth. Max-based selection better estimates target class proportions than mean-based selection, while deeper miners improve performance up to a similar result for three and four layers.
- Selection strategies: MineGAN (mean) backpropagates through all images, whereas MineGAN (max) backpropagates only through the image with the highest critic score.
- Selection strategies: The target distribution for Red vehicles is 0.3:0.7 for cars versus buses.
- Selection strategies: MineGAN (mean) equally selects the Car and Bus pretrained models, while MineGAN (max) estimates the target class distribution.
- Miner Architecture: Using more fully connected layers in the miner increases performance on the off-manifold Arch target class.
- Miner Architecture: Three- and four-layer miners produce similar results, suggesting that additional layers would provide limited benefit in this setting.
D. MNIST experiment
The MNIST experiments evaluate MineGAN with FID and classifier error for unconditional and conditional targeted generation. Mining identifies suitable latent regions, including for the more difficult off-manifold task.
- Evaluation: FID and classifier error evaluate unconditional and conditional MNIST generation at noise length |z| = 128.Classifier error measures misclassified generated images for the intended class.
- Visualization: Figure 10 visualizes unconditional off-manifold generation for digits 6, 4, 3, 2, 1, and 0.The figure complements the quantitative MNIST evaluation.
- Results: The miner identifies suitable regions for target digits, as indicated by relatively low evaluation errors.This conclusion holds for both unconditional and conditional targeted generation.
- Results: Conditional mining performs better than unconditional mining because it selects the target class more often.The comparison is reported for the quantitative MNIST results.
- Results: Off-manifold generation is more difficult than on-manifold generation, but mining still finds suitable regions by using local patterns shared with the target.The off-manifold scores remain reasonably low despite the higher evaluation scores.
E. Further results on CelebA
Additional CelebA experiments cover both on-manifold and off-manifold target domains using a pretrained Progressive GAN. The evaluations include women and children face-generation targets.
- On-manifold transfer: CelebA→FFHQ women provides an additional on-manifold experiment based on a pretrained Progressive GAN.The results are shown in Figure 11.
- Off-manifold transfer: CelebA→FFHQ children evaluates off-manifold transfer from CelebA to FFHQ using a pretrained Progressive GAN.The results are shown in Figure 12.
- On-manifold transfer: CelebA→CelebA women provides another on-manifold experiment based on a pretrained Progressive GAN.The results are shown in Figure 13.
F. Further results for LSUN
Further LSUN results examine transfer to red vehicles, towers, and bedrooms from one or multiple pretrained Progressive GAN domains. The figures also compare training from scratch, TransferGAN, and MineGAN.
- Vehicle transfer: The red-vehicle experiment transfers from the bus and car domains using a pretrained Progressive GAN.The results are shown in Figure 16.
- Multiple-source transfer: For multiple pretrained GANs, one source domain initializes the critic, with Church used for Tower and Kitchen for Bedroom.The authors report that this initialization choice had little influence on the final result.
- Women faces: Figure 14 compares 100 FFHQ women faces with training from scratch and TransferGAN, identifying images remembered by those models.The scratch-trained model starts at low quality and iteratively overfits to a particular training image.
- Children faces: Figure 15 shows 100 FFHQ children faces and marks images remembered by the model trained from scratch.The figure is based on a pretrained Progressive GAN.
- Unconditional transfer: The unconditional GAN experiments transfer livingroom, kitchen, bridge, and church domains to tower and bedroom targets.Figure 17 presents results for both target sets using a pretrained Progressive GAN.