Source-linked AI summary
Few-shot Image Generation with Elastic Weight Consolidation
Yijun Li, Richard Zhang, Jingwan Lu, Eli Shechtman
TL;DR
Few-shot image generation must adapt to a target domain from very few images without simply memorizing them or losing useful source diversity. The paper adapts a pretrained generator without adding parameters and regularizes weight changes using parameter importance. It reports effective, diverse generation across target domains, including extremely few-shot settings, while showing that adaptation depends on the amount of target data and source–target similarity.
Problem
Few-shot image generation must generalize from only a few target-domain images because limited data makes directly inferring the target distribution difficult.
Method
The method adapts a pretrained generative model's weights without additional parameters and selectively regularizes weight changes using Fisher-information-based importance.
Results
The method produces diverse target-domain generations from limited data and is evaluated across target-example counts and source–target dissimilarities, including extremely few examples.
Takeaways & Limitations
Source-domain diversity can be inherited while target appearance is adapted, but successful transfer is better supported when source and target domains are similar.
Takeaways & Limitations
Performance degrades when source and target domains are highly dissimilar; adapting faces to color-pencil landscapes can preserve facial silhouettes despite EWC regularization.
Abstract
from arXiv · showhide
Few-shot image generation seeks to generate more data of a given domain, with only few available training examples. As it is unreasonable to expect to fully infer the distribution from just a few observations (e.g., emojis), we seek to leverage a large, related source domain as pretraining (e.g., human faces). Thus, we wish to preserve the diversity of the source domain, while adapting to the appearance of the target. We adapt a pretrained model, without introducing any additional parameters, to the few examples of the target domain. Crucially, we regularize the changes of the weights during this adaptation, in order to best preserve the information of the source dataset, while fitting the target. We demonstrate the effectiveness of our algorithm by generating high-quality results of different target domains, including those with extremely few examples (e.g., <10). We also analyze the performance of our method with respect to some important factors, such as the number of examples and the dissimilarity between the source and target domain.
1 Introduction
Few-shot image generation adapts a pretrained model from an abundant source domain to a target domain with only a few examples. The method preserves source-domain diversity by selectively regularizing weight changes while fitting the target appearance.
- Motivation: Generative models struggle when practical applications provide only limited target-domain data, motivating generalization from a few new examples.The paper contrasts this setting with GAN demonstrations using 70,000 aligned-face images or 1.3M images across classes.
- Motivation: Few-shot generation leverages shared latent factors between source and target domains, such as extending face pose and expression variations to emojis.The source and target may differ in appearance while retaining transferable structure.
- Method: The proposed adaptation changes pretrained weights without adding parameters, avoiding manual design of new parameter modules.The adaptation must fit limited target data while retaining transferred knowledge and source diversity.
- Method: Parameter importance is estimated to preserve important weights during generative-model adaptation, drawing on Elastic Weight Consolidation.The paper notes that parameters should not be treated equally during tuning.
- Evaluation: The method evaluates multiple cross-domain source/target pairs and analyzes how target-example count and source–target dissimilarity affect performance.The experiments include artistic domains with limited data.
2 Related Work
Related work spans few-shot classification and generation, style transfer, and continuous learning. This paper differs by addressing diverse image generation from only a few target images while adapting across domains and preserving source diversity.
- Few-shot learning: Few-shot classification typically has few labeled instances but may have abundant unlabeled images, whereas few-shot generation assumes only a few target-domain images.Generation also seeks diverse outputs rather than a consistent semantic label.
- Few-shot learning: Prior generative approaches include matching, sequential, and autoregressive models, but some are limited to simple patterns and low-resolution results.Recent GAN-based work showed promising high-resolution results on complex natural images.
- Style transfer: Style-transfer methods generate target-domain data by transferring example styles to abundant source data, but example-based methods can require alignment or transfer only color and texture.The paper distinguishes example-based and domain-based style transfer.
- Continuous learning: Continuous-learning methods address catastrophic forgetting across sequential tasks, but prior generative studies assume sufficient data for each task.This work instead adapts with only a few target examples and seeks to retain source-domain diversity rather than source-domain generation itself.
3 Proposed Method
The method adapts a pretrained generative model to a few target examples without adding parameters, while regularizing weight changes to preserve source-domain diversity. It uses layer- and parameter-specific importance, measured through weight-change analysis and Fisher information, to balance target fitting against source preservation.
- Motivation: Direct adaptation without weight regularization overfits few target examples, often re-generating near-replicas instead of learning the target domain.The problem arises because the model has far more parameters than target examples.
- Rate of changes on weights: Weights in the generator’s last layer change least across source-to-target adaptation, suggesting that some last-layer weights should be preserved more strongly.This pattern is also observed across LapGAN and StyleGAN with other source-target pairs.
- Importance measure: The method estimates parameter importance with Fisher information computed from generated source-domain data and a discriminator-based log-likelihood.The discriminator output is used as an equivalent binary crossentropy loss for the log-likelihood.
- Effectiveness of regularization: Adding EWC preserves source variation in 10-shot Fernand Léger generation, whereas removing it causes rapid weight deviation and results close to reproducing the given examples.The ablation sets λ = 0 and observes rapid changes within a few hundred iterations.
- Importance measure: The EWC loss penalizes changes to important parameters relative to source weights, with λ balancing the regularization and adversarial adaptation losses.The parameter importance weights determine how strongly each change is penalized.
4 Experimental Results
Experiments compare few-shot adaptation methods across source–target domains and analyze how performance changes with target-shot count, regularization strength, and domain dissimilarity. The proposed method generally preserves target-style fidelity and diversity, but its success depends on sufficient similarity between source and target domains.
- Experimental settings: Experiments compare StyleGAN-based adaptation with NST, BSA, and MineGAN across face and landscape source–target pairs.The evaluation includes qualitative and quantitative comparisons, using 10-shot examples and analyses of several adaptation factors.
- Qualitative results: The proposed method produces target-style images with greater diversity than competing adaptation methods in visual comparisons.NST mainly transfers color and texture, BSA shows blur and mode collapse, and MineGAN can regenerate training examples under few-shot adaptation.
- Quantitative comparisons: The proposed method achieves the lowest FID among the compared schemes and generates more diverse results than BSA and MineGAN in the 10-shot study.The user study also reports the highest fooling rate for the proposed method, indicating the strongest reported realism in the evaluated 10-example target domains.
- Number of shots: With more target examples, the proposed method and MineGAN improve similarly, while the proposed method has a clearer advantage over MineGAN when the target has ≤10 examples.NST changes little with additional examples, whereas BSA performance drops as the number of examples increases.
- Regularization weight λ: The regularization weight λ controls a trade-off: larger values preserve source diversity but hinder target adaptation, while smaller values increase overfitting risk and reduce diversity.The analysis identifies balancing source preservation against target fitting as an unavoidable design trade-off.
- Source–target dissimilarity: Adaptation quality decreases as source–target dissimilarity increases, because EWC cannot sufficiently change the semantic shape when adapting faces to landscapes.Results are most realistic and diverse for targets close to FFHQ and become unrealistic for highly dissimilar targets; the authors recommend selecting a similar pretraining source.
5 Conclusion
The method adapts a pretrained generative model to few-shot target domains while preserving source-domain diversity and avoiding overfitting. Experiments across domains show favorable performance and potential utility for expanding scarce datasets.
- The method adapts a pretrained generative model to a new target domain without adding parameters.It selectively regularizes weight changes using estimated parameter importance.
- It inherits source-domain diversity while adapting to the target appearance from only a few examples.
- The method performs favorably against existing methods across various domains and helps expand data-scarce domains for downstream image synthesis.
Broader Impact
The work aims to support creative applications by expanding limited datasets for large-scale image-synthesis pipelines. It also addresses deception risks through detectability analysis, while acknowledging that content authenticity remains unresolved.
- Generated data may facilitate large-scale training in creative image-synthesis pipelines where manually creating datasets is laborious.
- Detectability: A detector achieved 94.9% AP on Cat and 99.6% AP on CelebA-Female generated images.
- Detectability: Detection performance is incomplete and may degrade after compression or rescanning, leaving content authenticity as a significant challenge.