Source-linked AI summary
Few-Shot Unsupervised Image-to-Image Translation
Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, Jan Kautz
TL;DR
Existing unsupervised image-to-image translation methods require many images from target classes during training and do not support few-shot generalization to unseen classes. FUNIT combines adversarial training with a network that extracts class appearance from a few test-time examples, achieving strong translation results while remaining limited to visually related novel classes.
Problem
Existing methods require large training sets over all classes they translate and cannot generalize to new classes from few examples.
Method
FUNIT trains a GAN-based translator on many source classes, using a content image and a few target-class images to generate analogous target-class outputs.
Results
FUNIT outperformed baselines across metrics on Animal Faces and North American Birds, achieving 82.36 and 96.05 Top-5 (test) accuracy in 1-shot and 5-shot Animal Faces settings.
Takeaways & Limitations
Few-shot translation performance improves with more training classes and more target-class shots, and generated images can also support few-shot classification.
Takeaways & Limitations
FUNIT fails when novel object appearances differ dramatically from source classes, often producing color-changed versions of the input.
Abstract
from arXiv · showhide
Unsupervised image-to-image translation methods learn to map images in a given class to an analogous image in a different class, drawing on unstructured (non-registered) datasets of images. While remarkably successful, current methods require access to many images in both source and destination classes at training time. We argue this greatly limits their use. Drawing inspiration from the human capability of picking up the essence of a novel object from a small number of examples and generalizing from there, we seek a few-shot, unsupervised image-to-image translation algorithm that works on previously unseen target classes that are specified, at test time, only by a few example images. Our model achieves this few-shot generation capability by coupling an adversarial training scheme with a novel network design. Through extensive experimental validation and comparisons to several baseline methods on benchmark datasets, we verify the effectiveness of the proposed framework. Our implementation and datasets are available at https://github.com/NVlabs/FUNIT .
1. Introduction
Existing unsupervised image-to-image translation methods require many training images for every class and cannot generalize to new classes from few examples. FUNIT addresses this gap by learning from many source classes and using a few target-class images at test time.
- Existing translation methods transfer complex appearance changes but require large training sets for every class and lack few-shot generalization.
- FUNIT maps source-class images to analogous images of previously unseen target classes using only a few target examples provided at test time.The target class is not represented during training.
- FUNIT trains on images from many object classes to learn appearance patterns that generalize to unseen classes.The approach is motivated by the hypothesis that broader past visual experience supports few-shot generation.
- The framework couples adversarial training with a novel network design to achieve few-shot unsupervised image-to-image translation.
- Experiments on three datasets with multiple metrics verify the framework and show its generated images can support few-shot classification.A classifier trained on generated images outperformed a state-of-the-art feature-hallucination method.
2. Related Work
Unsupervised image-to-image translation is an ill-posed problem addressed with additional constraints, but existing methods remain sample-inefficient and task-specific. Related multi-class and few-shot methods differ from FUNIT in their data assumptions or focus on translating among seen classes.
- Unsupervised image-to-image translation learns mappings between unpaired source and target classes, but recovering the joint distribution from marginals is inherently ill-posed.Existing approaches add constraints such as property preservation, cycle consistency, or shared/paired representations.
- Existing models generate poor translations with few training images and cannot directly transfer a trained translation model to a new class pair.
- Benaim and Wolf use one source image and many target images, whereas FUNIT assumes many source images and few target images.
- Multi-class translation methods train on multiple classes but translate among seen classes, while FUNIT translates seen source classes to previously unseen target classes.
- Few-shot classification methods use shared generative priors, meta-learning, embeddings, or feature generation; FUNIT targets few-shot unsupervised translation but can also be applied to classification.
3. Few-shot Unsupervised Image Translation
FUNIT translates a content image into an unseen target class using a few target examples available only at test time. Its encoder–decoder design separates content structure from class appearance, while adversarial and auxiliary losses support training and generalization.
- Few-shot translation: FUNIT uses many source-class images during training and a few unseen target-class images at test time to translate source images into the target class.The same model can be reused with different novel target classes.
- Few-shot translation: The generator takes a content image and K target-class images, assuming the content and class images belong to different object classes.The target examples specify the desired class at test time.
- Network design: The generator combines a content encoder, class encoder, and decoder, with the class encoder averaging latent vectors from the K class images.The content encoder produces a spatial content code, while the class encoder produces a class code.
- Network design: AdaIN layers inject class-code parameters into the decoder so class images control global appearance while the content image determines local structure.The design aims to separate class-invariant factors such as pose from class-specific factors such as appearance.
- Results and learning: Generalization improves with more source classes, and FUNIT outperforms fair and unfair baselines across reported metrics on Animal Faces and North American Birds.The framework extends reconstruction and feature-matching losses to few-shot unsupervised translation; Table 1 compares fair and unfair baselines, with higher or lower values preferred as indicated.
- Adversarial learning: The multi-task discriminator performs separate binary real-versus-generated classification tasks for each source class instead of one harder multi-class task.Generator and discriminator updates use the binary score corresponding to the relevant class.
4. Experiments
Experiments evaluate FUNIT’s few-shot translation across datasets, baselines, metrics, visual comparisons, and downstream classification. FUNIT performs strongly when novel classes resemble training classes, while performance degrades for visually dissimilar classes.
- Experimental setup: FUNIT trains with K = 1 target image and is evaluated on Animal Faces, birds, and other datasets using few-shot translation protocols.The evaluation samples target-class images at test time and compares fair baselines without target images during training against unfair baselines with target images.
- Quantitative results: 82.36 and 96.05 Top-5 (test) accuracy are achieved in the 1-shot and 5-shot Animal Faces settings, versus 60.19 and 75.75 on North American Birds.FUNIT outperforms fair baselines across reported metrics and, with 5 shots, surpasses unfair baselines trained with 20 target-class images.
- Quantitative results: Increasing target images from K = 1 to K = 5 improves FUNIT across all metrics, with the largest boost occurring between those settings.The StarGAN-Fair baseline does not show the same trend.
- Qualitative results: FUNIT outputs preserve input poses while producing photorealistic images resembling novel target classes in the few-shot setting.Visual comparisons report that competing baselines either produce artifacts or reproduce the input content image.
- Analysis and limitations: FUNIT’s performance is positively correlated with the number of source classes, but it fails when unseen classes differ dramatically in appearance from source classes.In such cases, outputs tend toward color-changed versions of the input content images; shape changes are also absent in the AdaIN comparison.
- Downstream evaluation: FUNIT-generated images improve few-shot classification over feature hallucination, achieving around 2% improvement over the S&H method.Both methods outperform a classifier trained from only the single provided real image per novel class.
5. Discussion and Future Work
FUNIT introduced few-shot unsupervised image-to-image translation for unseen object classes, using a few target examples at test time. Its success depends on learned class-invariant and class-specific representations and is limited when novel classes differ dramatically in appearance from training classes.
- FUNIT introduced a few-shot unsupervised image-to-image translation framework for previously unseen object classes.
- The framework uses a source image and a few test-time examples from an unseen target class to generate a corresponding target-class image.
- FUNIT depends on class-invariant content encoding, class-specific target encoding, and generalization of the class encoder to unseen classes.
- FUNIT works more readily when novel classes are visually related to the source classes.
- When novel classes differ dramatically in appearance, FUNIT fails to translate effectively and tends to produce color-changed versions of the input.
A. Network Architecture
The translator combines class-invariant content information with class-specific information extracted from a few target images. A three-subnetwork design uses a content encoder, class encoder, and decoder, with AdaIN blocks controlling content decoding through class-derived affine parameters.
- The translator contains a content encoder, class encoder, and decoder.
- The content encoder maps the input image to a feature map intended to represent class-invariant content, including part locations rather than class-specific appearance.
- The class encoder maps K target-class images to class-specific latent vectors, which are elementwise averaged into one class latent code.
- The decoder combines the class latent code with the content latent code to generate the translation output.
- AdaIN residual blocks use decoder-produced scaling factors and biases, applying the same affine transformation at every spatial location to control content decoding.
- The discriminator is a Patch GAN architecture using Leaky ReLU, no normalization, convolutional layers, residual blocks, and downsampling.
B. Performance Metrics
The evaluation measures translation accuracy, content preservation, image quality, and distribution matching. Accuracy uses two classifiers and four Top-1/Top-5 metrics, while DIPD and FID assess preservation and distributional similarity.
- Translation accuracy is measured by whether classifiers recognize outputs as belonging to the target class.
- The evaluation reports Top1-all, Top5-all, Top1-test, and Top5-test accuracy metrics.
- Content preservation is quantified with DIPD using instance-normalized VGG conv5 features from input and translated images.
- FID compares translated-image distributions using activations from the last average-pooling layer of ImageNet-pretrained Inception-V3.
- Larger source-class training sets improve few-shot translation performance on the North American Birds dataset.
D. Ablation Study
Ablations show trade-offs among reconstruction weighting and loss components, while training generally improves several evaluation measures. Compared with AdaIN style transfer, the proposed method changes shapes as well as textures in animal-face translation.
- λR = 0.1 provides a good trade-off between content preservation and translation accuracy and is used as the default.
- Increasing λR reduces domain-invariant perceptual distance but lowers translation accuracy, whereas λR = 0.01 degrades both measures.
- Removing feature matching slightly degrades performance, while removing the zero-centered gradient penalty substantially degrades content preservation and translation accuracy.
- In the one-shot setting, translation accuracy, content preservation, image quality, and distribution matching generally improve with more training iterations, slowing around 10000 iterations.
- Unlike AdaIN style transfer, which changes textures without changing shapes, the proposed method addresses shape as well as appearance in animal-face translation.
F. Failure Case
The proposed model has documented failure cases, while accompanying studies examine sensitivity, ablations, training progress, and few-shot translation comparisons.
- Failure cases: The model’s typical failures include generating hybrid classes, ignoring input content images, and ignoring input class images.
- Sensitivity analysis: The content reconstruction loss weight λR is studied for its trade-off between content preservation and translation accuracy, with 0.1 used by default.
- Ablation study: Removing feature matching or gradient penalty losses generally worsens performance across evaluation criteria relative to the default setting.
- Training analysis: Few-shot translation performance is evaluated as a function of training iterations on the Animal Faces and North American Birds datasets.
- Comparisons: FUNIT-1 is compared with AdaIN style transfer for few-shot image translation, alongside one-shot classification evaluations on two datasets.
G. Latent Space Interpolation
The class encoder organizes related classes near one another and supports interpolation between class representations, sometimes producing an unseen intermediate class.
- Class embedding: Images from similar classes are grouped together in the learned class embedding space.
- Interpolation: Keeping content fixed while interpolating class codes between Siamese cat and Tiger can sometimes generate the previously unseen Tabby cat class.
- Interpolation: The interpolation result suggests that the class encoder learns a general class-specific representation that supports generalization to novel classes.
H. Few-Shot Classification
The paper evaluates generated images for one-shot classification and reports improvements over real-image-only baselines and feature-generation comparisons, including compatibility with prototypical networks.
- Setup: The one-shot classification experiment uses five training splits on Animal Faces and North American Birds, with one image per test class for training.
- Setup: FUNIT generates synthetic training images by combining novel-class images as class inputs with randomly sampled source-class images as content inputs.
- Results: Both FUNIT and Shrink and Hallucinate outperform a classifier trained with only one real image per novel class.
- Results: Around 2% improvement over Shrink and Hallucinate is obtained using generated images on two challenging fine-grained classification tasks.
- Evaluation protocol: The classifier experiments learn only a new classifier layer from features extracted by a network trained on the generator’s training classes.
- Results: Using generated samples with the one provided sample improves Prototypical Networks accuracy on both datasets by over 5.5%.
I. More Translation Results
Additional experiments visualize few-shot translation across animal, bird, flower, food, and face tasks, while a fully trained variant outperforms several established translation models.
- Additional visualizations: Additional few-shot translation results cover animal face, bird, flower, food, and celebrity face tasks using FUNIT-5.
- Standard setting: FUNIT-Unfair-All outperforms CycleGAN, UNIT, MUNIT, and StarGAN when all animal-face training images and target classes are available during training.
- Additional visualizations: The animal-face visualizations use five target-class images at test time to specify a dynamically selected target class.
- Additional visualizations: The bird visualizations likewise use five target-class images at test time for dynamically specified target-class generation.
- Additional visualizations: Flower, food, and face translation visualizations are also computed with FUNIT-5 across dynamically specified target classes.