Source-linked AI summary
One-Shot Unsupervised Cross Domain Translation
Sagie Benaim, Lior Wolf
TL;DR
The paper asks how to translate a single, potentially unseen image from domain A into domain B, a challenge for constructing an analogous image from a sample unlike those previously observed. It adapts a variational autoencoder for B to fit the single A image. OST outperforms existing algorithms in the low-shot scenario and, on most datasets, reaches comparable accuracy using one A example to methods trained on all A images.
Problem
How can an analogous image be generated from a single, potentially unseen domain A sample when no other A training images are available?
Source: §1 · p. 1Method
OST trains a variational autoencoder for domain B, then adapts cloned autoencoders using reconstruction and one-way circularity losses, with the single A sample updating only unshared layers.
Source: §1 · p. 2Results
OST outperforms existing algorithms in the low-shot scenario and, on most datasets, achieves comparable accuracy with one A training example to other methods using all A images.
Source: §1 · p. 2Takeaways & Limitations
The one-shot setting removes constraints on inter-sample correlations in domain A, offering a perspective on mechanisms underlying unsupervised domain translation.
Source: §1 · p. 2Phase II adapts domain B’s autoencoder to domain A to translate x, while multiple one-shot domain adaptations would require a more conservative approach.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Given a single image x from domain A and a set of images from domain B, our task is to generate the analogous of x in B. We argue that this task could be a key AI capability that underlines the ability of cognitive agents to act in the world and present empirical evidence that the existing unsupervised domain translation methods fail on this task. Our method follows a two step process. First, a variational autoencoder for domain B is trained. Then, given the new sample x, we create a variational autoencoder for domain A by adapting the layers that are close to the image in order to directly fit x, and only indirectly adapt the other layers. Our experiments indicate that the new method does as well, when trained on one sample x, as the existing domain transfer methods, when these enjoy a multitude of training samples from domain A. Our code is made publicly available at https://github.com/sagiebenaim/OneShotTranslation
1 Introduction
The paper frames one-shot translation as a version of perceptual blending: an agent must construct an analog for a sample it has not previously seen. It introduces OST, which adapts a domain-B variational autoencoder to a single domain-A sample, and reports strong results against methods trained with more domain-A data.
1 Introduction
- The task is motivated by perceptual blending: an agent generates an analog of a newly observed sample and compares it with the observation to update its internal model.Source: §1 · p. 1
- The task asks for an image analogous to a single sample from an unknown domain A, using many samples or a model of domain B, a setting the authors say prior literature had not addressed.Source: §1 · p. 1
- The authors clarify that the one-shot setting provides no domain-A training images beyond x, unlike zebra-to-horse translation with many zebra examples.Source: §1 · p. 1
- OST first constructs a variational autoencoder for domain B, then adapts a cloned autoencoder using the single sample x while restricting its gradient updates to unshared layers.The method also augments domain B with slight rotations and random horizontal translations, and applies augmentation to x for stability.Source: §1 · pp. 1–2
- OST outperforms existing algorithms in the low-shot scenario and, on most datasets, matches methods trained on all domain-A images using only one example.The paper links this result to the absence of inter-sample correlation constraints in domain A in the one-shot case.Source: §1 · p. 2
2 Previous Work
Prior unsupervised translation work uses several cues to infer mappings, including shared objects, preserved relations, shared autoencoder structure, and architectural constraints. In the one-shot setting, the authors identify inner-object structure as the only available cue and report that their own image-part comparison attempts did not work.
2 Previous Work
- Unsupervised domain translation learns mappings from two unpaired, unlabeled datasets, one from each domain.Source: §2 · p. 2
- Prior work constrains mappings through shared objects, pairwise object relations, inner-object part arrangements, shared autoencoder weights, and network, GAN, and circularity constraints.Source: §2 · p. 2
- Figure 1 depicts two phases: training a domain-B variational autoencoder, then cloning it with shared encoder and decoder parts frozen with respect to x; the caption also defines reconstructed and translated sample spaces.Source: Figure 1 · p. 3, §2 · p. 3
- In one-shot translation, the domain-invariant distance is undefined and domain A has no inter-object relations, leaving inner-object structure as the only available cue.Source: §2 · pp. 2–3
- The authors’ attempts to compare image parts within x failed; their approach instead assumes domain A and B have similar mid-level representations.Source: §2 · p. 3
3 One-Shot Translation
OST first trains a domain-B variational autoencoder, then adapts a domain-A copy using the single sample while selectively updating unshared layers. Shared encoder and decoder layers provide a common structure, and training also uses domain-B adversarial feedback.
3 One-Shot Translation
- The encoders share their top layers and the decoders their bottom layers, enforcing a common encoding structure; a domain-B discriminator supplies adversarial feedback.The method uses a domain-B discriminator because low-shot domain A lacks enough real examples and the translation direction is from A to B.Source: §3 · p. 3
3.1 Phase One of Training
- Phase one trains a domain-B variational autoencoder on randomly augmented images using reconstruction, variational, and adversarial losses.Augmentation applies small random rotations and horizontal translations; the discriminator is trained separately from the generator.Source: §3.1 · p. 4
3.2 Phase Two of Training
- Phase two adds reconstruction on x, a one-way cycle loss, and adversarial losses for translating x to domain B.The shared encoder and decoder are adapted indirectly through domain-B training; losses outside LI update only unshared layers, while LI updates both shared and unshared layers.Source: §3.2 · pp. 4–5
- Selective backpropagation adapts domain-A unshared layers while protecting shared layers from overfitting to the single sample; those shared layers still adapt indirectly.Fixing the shared representation entirely hurts performance, while updating it directly from the one sample can produce unstable, arbitrary mappings.Source: §3.2 · p. 5
3.3 Network Architecture and Implementation
- OST, CycleGAN, and UNIT use the same CycleGAN architecture for fair comparison, with convolutional, residual, and deconvolutional layers.For MNIST–SVHN experiments, the paper uses DCGAN rather than PatchGAN; Figure 3 plots classifier accuracy against the log-scaled number of domain-A samples in both directions.Source: §3.3 · pp. 5–6, Figure 3 · p. 6
- Without selective backpropagation, Figure 2 shows the SVHN-to-MNIST result becoming unstable and arbitrary across training iterations.Figure 3 compares OST, UNIT, and CycleGAN using pretrained-classifier accuracy for translations in both directions.Source: Figure 2 · p. 5, Figure 3 · p. 6
4 Experiments
The experiments compare OST with UNIT and CycleGAN in one-sample and larger-data settings across digit, style-transfer, and drawing tasks. OST outperforms baselines on digit accuracy, matches full-training-set baselines on drawing metrics, and selective backpropagation has the largest ablation effect.
4 Experiments
- The experiments compare OST trained on one A-domain sample with UNIT and CycleGAN trained on that sample or the full A-domain set, reporting mean results across repeated runs.Evaluations include style and content losses and classifier-based target-image accuracy.Source: §4 · p. 6
MNIST to SVHN Translation
- OST outperforms the baselines on MNIST-to-SVHN translation accuracy even when they use the entire A-domain training set.Accuracy rises as the number of A-domain samples increases; it is measured on training images translated from A to B, not a separate test set.Source: §4 · p. 6
- Selective backpropagation has the largest effect on one-shot translation accuracy; one-way cycle loss and augmentation contribute less.Table 1 reports the ablation for one-sample MNIST-to-SVHN translation and the reverse.Source: §4 · p. 6, Table 1 · p. 7
Style Transfer Tasks
- For style transfer, OST captures target style similarly to UNIT and CycleGAN trained with many samples and CycleGAN trained with one, whose high perceptual distance indicates poorer content preservation.Low perceptual distance indicates that more high-level content is preserved.Source: Table 2 · p. 7, §4 · p. 7
MNIST to SVHN Translation
- Freezing the shared encoder and decoder in phase II fails to produce images in the target distribution, such as gray rather than black backgrounds in SVHN-to-MNIST translation.Source: §4 · p. 7
Drawing Tasks
- On drawing tasks, one-sample OST yields aligned mappings and achieves perceptual distance and style difference similar to full-training-set UNIT and CycleGAN.The evaluation also reports user-study correctness; the supplied passages give no specific values.Source: §4 · pp. 7–8, Table 3 · p. 8
5 Discussion
OST is framed as useful for agents encountering unseen domains, but its phase-II adaptation is geared toward translating the current sample. The discussion identifies multi-one-shot adaptation and the reverse transfer direction as open boundaries.
5 Discussion
- Phase II adapts domain B to translate x, so multi-one-shot domain adaptation would require a more conservative approach.The authors frame the method as suitable for agents encountering images from unseen domains.Source: §5 · p. 8
- The demonstrated direction is one sample from unseen domain A to B; the reverse direction, from familiar A to a new B with one sample, remains an open question.An analogous two-phase approach is proposed, but adversarial training cannot be applied directly to the target domain with only one observed sample.Source: §5 · p. 8
- For reverse-direction transfer, the authors suggest the new domain might still be modeled from variability in the familiar source domain.This possibility is raised alongside the limitation on direct adversarial training with a single target sample.Source: §5 · p. 8