Source-linked AI summary
Few-shot Image Generation via Cross-domain Correspondence
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, Richard Zhang
TL;DR
Training generative models with very few target images can cause overfitting, motivating transfer from a large source domain. The paper preserves source-instance relationships with a cross-domain distance consistency loss and uses anchor regions with different realism constraints. Experiments show correspondence discovery and diverse, realistic generations, while examples reveal that some learned correspondences remain imperfect.
Problem
Generative models trained on target domains with very limited examples can overfit, although many domains contain only a few images.
Method
The paper adapts a pretrained source GAN using cross-domain distance consistency and anchor-based realism constraints across latent-space regions.
Results
The model automatically discovers correspondences between source and target domains and generates diverse and realistic images across related domains.
Takeaways & Limitations
Transferring relationships between source instances provides a data-efficient route to few-shot generation while preserving diversity and realism.
Takeaways & Limitations
Some correspondences remain imperfect, producing artifacts such as red cars turning orange or blonde hair becoming dark with sunglasses.
Abstract
from arXiv · showhide
Training generative models, such as GANs, on a target domain containing limited examples (e.g., 10) can easily result in overfitting. In this work, we seek to utilize a large source domain for pretraining and transfer the diversity information from source to target. We propose to preserve the relative similarities and differences between instances in the source via a novel cross-domain distance consistency loss. To further reduce overfitting, we present an anchor-based strategy to encourage different levels of realism over different regions in the latent space. With extensive results in both photorealistic and non-photorealistic domains, we demonstrate qualitatively and quantitatively that our few-shot model automatically discovers correspondences between source and target domains and generates more diverse and realistic images than previous methods.
1. Introduction
Few-shot target domains are difficult for generative models because very limited data encourages overfitting. The paper transfers relational diversity from a large source domain through cross-domain consistency and realism controls, producing diverse and realistic target images.
- Motivation: Generative models typically require thousands of images, making domains with only 10 examples per artist challenging to model.The Artistic-Faces dataset exemplifies this limited-data setting.
- Motivation: Existing transfer-learning methods preserve source information or augment data, but many target settings contain more than 100 images and extreme few-shot adaptation often overfits.The related-work passage describes overfitting when only a few target images are available.
- Method: The proposed cross-domain distance consistency loss transfers how instances relate to one another by matching pairwise-distance distributions before and after adaptation.This adapts models rather than translating source images.
- Results: For related domains, the approach automatically discovers one-to-one correspondences and more faithfully models target diversity and realism; for unrelated domains, it still finds part-level correspondences.The unrelated-domain case does not model the target distribution but can generate diverse samples.
- Method: An image-level adversarial loss is applied to synthesized images corresponding to real samples, while other samples receive only patch-level adversarial supervision.This allows only a subset of generations to resemble the few-shot examples directly, while the rest capture patch-level texture.
- Results: Extensive qualitative and quantitative experiments show that the framework discovers source-target correspondences and generates diverse, realistic images.The stated contribution covers both related source-target domains and few-shot image generation.
2. Related work
Related work addresses few-shot generation through model adaptation, domain translation, and distance preservation. This paper instead adapts a source generator using unlabeled source data and cross-domain relational consistency.
- Few-shot learning: Few-shot classification commonly learns feature similarity between query and support examples or learns to adapt a base learner to a new task.These approaches address classification rather than unconditional image generation.
- Few-shot learning: Few-shot image generation seeks diverse new examples while avoiding overfitting, commonly adapting a model pretrained on a large source domain.Existing methods add parameters or update source parameters with regularization.
- Domain translation: Domain-translation methods are unsuitable for the few-shot setting because they require substantial source and target data, and some recent methods require labeled data.The paper assumes large unlabeled source data and adapts the source model for unconditional generation.
- Distance preservation: Distance-preservation methods improve generation diversity by preserving distances between paired inputs and outputs; this work applies distance consistency across source and target domains.The stated goal is to inherit diversity learned by the source model.
3. Approach
The approach adapts a pretrained source generator to a few-shot target by preserving cross-domain structure and applying region-dependent realism constraints. These components are designed to prevent overfitting while retaining diversity and correspondence between source and target generations.
- Source-to-target adaptation: The method initializes an adapted generator from a source generator trained on a large dataset and fits it to a small target dataset.The source generator maps noise vectors into images, while adaptation starts from its pretrained weights.
- Source-to-target adaptation: Naive adversarial adaptation overfits in extreme few-shot settings because the discriminator can memorize the limited target examples.This causes generated outputs to collapse toward the few-shot training images.
- Cross-domain distance consistency: The cross-domain distance consistency loss preserves the distribution of pairwise distances between source and adapted generated samples.It uses similarities across generated instances and feature layers, with KL-divergence encouraging the adapted model to match source distributions.
- Relaxed realism: Relaxed realism applies a full-image discriminator in anchor latent regions and a patch-level discriminator elsewhere.This limits strict matching to a subset of generated samples while allowing the remaining samples more structural freedom.
- Relaxed realism: Anchor regions are formed from k fixed random latent points, with small Gaussian perturbations, and the patch discriminator shares weights with the image discriminator.The perturbation uses σ = .05, while patch scale depends on the network architecture and selected layer.
- Final objective: The final objective combines target-appearance adversarial loss with distance consistency to preserve structural diversity from the source model.The patch discriminator provides additional freedom, while the consistency loss directly leverages the source generator.
4. Experiments
Experiments evaluate few-shot adaptation against baselines on quality, diversity, component effects, correspondence, and target-data scale. Across these settings, the method produces more diverse and realistic generations while preserving source–target correspondences, including part-level mappings for unrelated domains.
- Quality and Diversity Evaluation: The method is compared with TGAN, BSA, MineGAN, FreezeD, and EWC across multiple source and target domains.Source models include FFHQ, Church, Cars, and Horses; targets include caricatures, sketches, paintings, babies, sunglasses, landscapes, haunted houses, Van Gogh houses, and abandoned cars.
- Quality and Diversity Evaluation: Compared with baselines, the method better captures caricature and sketch distributions, generating diverse and realistic images beyond the few training examples.Baselines either overfit or introduce only minor variations within a few modes, whereas the method can generate objects absent from training images, such as hats in sketches.
- Quality and Diversity Evaluation: The method significantly outperforms all baselines on FID for Babies and Sunglasses, while FID alone does not reflect overfitting in limited-data domains.Evaluation samples 10 images for training but uses the full datasets to measure modeling of the true distribution.
- Quality and Diversity Evaluation: Higher average intra-cluster LPIPS distances indicate that the method consistently generates more visually distinct images.The metric measures pairwise LPIPS distances among generated images assigned to the same training-image cluster.
- Quality and Diversity Evaluation: Removing Ldist reduces diversity, while using only Dimg or only Dpatch causes mode collapse or poorer image quality; combining them yields diverse, realistic generations.The full method preserves diversity at the part level and realism at both part and image levels.
- Analyzing source ↔target correspondence: For related domains, the same noise vectors produce clear correspondences, while unrelated domains generally fail to model the target distribution accurately but retain part-level mappings.Examples include Church structures mapping to caricature features, car wheels and bumpers mapping to caricature eyes and mouths, and horse legs mapping to haunted-house doors.
- Effect of target dataset size: Increasing the target set from 1 to 5 to 10 images increases diversity and adds details in generated sketches and landscapes.One-shot generations introduce small variations; five samples yield distinct identities; ten samples produce more details and more diverse landscapes.
5. Conclusion and Limitations
The method adapts pretrained GANs to small target domains by discovering cross-domain correspondences. Its limitations include correspondence errors that can reduce diversity, although the authors report progress toward data-efficient generation.
- Conclusion: The framework adapts a pretrained GAN to a small target domain by discovering cross-domain correspondences.
- Limitations: Correspondence errors can alter attributes incorrectly, such as changing a red car to orange or blonde hair to dark hair.
- Limitations: The authors identify better correspondence discovery as necessary for more diverse generations.
- Conclusion: The work demonstrates that existing source models can be leveraged to model new distributions with less data.
Supplementary
The supplementary material details architectures, baselines, experiments, and evaluation procedures. It reports qualitative correspondence behavior, baseline overfitting in the 1-shot setting, and separate diversity and quality assessment.
- Training and architecture: The generator and image discriminator use StyleGAN2 architectures, while the patch discriminator uses the first l layers of the image discriminator.
- Baselines: Reducing MineGAN’s miner-network complexity alone did not improve extreme few-shot results, with FID values of 96.72 for babies and 68.67 for sunglasses.
- FFHQ experiments: Adapting FFHQ preserves diversity in generated target-domain images through correspondence between source and target distributions.
- FFHQ experiments: FFHQ-to-caricature adaptation best embeds and reconstructs unseen caricature images, indicating caricature is most related to FFHQ among the tested sources.
- Baseline comparison: FreezeD and EWC overfit in the 1-shot setting by generating virtually identical sketches or scenes.
- Diversity assessment: The clustering assessment compares each method’s closest and farthest generated samples to training-image cluster centers using LPIPS.
- Additional experiments: The hand-gesture experiment adapts a source model trained on gestures over a plain surface to domains including landscapes, fire, and maps.
- Evaluation metrics: Density measures fake samples in real-sample neighborhoods, while coverage measures the fraction of real samples whose neighborhoods contain fake samples.