Source-linked AI summary
ComboGAN: Unrestrained Scalability for Image Domain Translation
Asha Anoosheh, Eirikur Agustsson, Radu Timofte, Luc Van Gool
TL;DR
Existing image translation systems are largely limited to two domains, making broader domain coverage costly because pairwise models scale quadratically. ComboGAN decouples domain encoders and decoders around a common representation, allowing mix-and-match translation with linear growth in models and training time while retaining comparable visual quality in the reported experiments.
Problem
Two-domain image translation systems do not scale efficiently to many domains because pairwise training requires a quadratic number of models.
Method
ComboGAN assigns an encoder and decoder to each domain, combining them so images can be translated through a shared representation into any other domain.
Results
ComboGAN produced reasonably good four-season translations and comparable quality to CycleGAN while training four networks instead of twelve.
Takeaways & Limitations
The framework maintains CycleGAN’s visual richness beyond two domains while allowing additional domains to be appended by training a new encoder/decoder pair alongside a pretrained model.
Takeaways & Limitations
The work omits encoder-decoder layer sharing and latent-space losses to isolate the effect of scaling the CycleGAN model.
Abstract
from arXiv · showhide
This year alone has seen unprecedented leaps in the area of learning-based image translation, namely CycleGAN, by Zhu et al. But experiments so far have been tailored to merely two domains at a time, and scaling them to more would require an quadratic number of models to be trained. And with two-domain models taking days to train on current hardware, the number of domains quickly becomes limited by the time and resources required to process them. In this paper, we propose a multi-component image translation model and training scheme which scales linearly - both in resource consumption and time required - with the number of domains. We demonstrate its capabilities on a dataset of paintings by 14 different artists and on images of the four different seasons in the Alps. Note that 14 data groups would need (14 choose 2) = 91 different CycleGAN models: a total of 182 generator/discriminator pairs; whereas our model requires only 14 generator/discriminator pairs.
1. Introduction
Image translation methods such as CycleGAN produce realistic, unsupervised semantic transfers but are generally trained for only two domains. ComboGAN addresses the resulting scalability problem by recombining domain-specific components into a shared representation, reducing model growth from quadratic to linear.
- Two-domain image translation methods can preserve semantic content and realism without paired examples, but require training data from two specific domains.
- Naively training one model for every desired domain pair requires Θ(n^2) models for n domains.
- ComboGAN divides each model into an encoder to a common representation and a decoder back into a target domain.
- Pairing one encoder and decoder per domain enables translation between arbitrary domains through a shared representation.
- Related approaches include paired conditional translation, UNIT with shared latent vectors and additional losses, and StarGAN with shared models across domains.
- CycleGAN uses adversarial and cycle-consistency losses to support unsupervised image translation while encouraging mappings between two domains to be inverses.
2. The ComboGAN Model
ComboGAN decouples domain-specific encoders and decoders so they can be recombined for translation among N domains. Its training samples domain pairs while keeping model and training growth linear in the number of domains.
- Model design: ComboGAN splits each CycleGAN generator into domain-specific encoder and decoder components, decoupling domains and networks.
- Model design: Any input domain can be translated to any other by applying the source encoder followed by the target decoder.The encoded representation can be cached when translating the same input into multiple target domains.
- Model design: One encoder-decoder generator per domain makes the generator count scale linearly rather than quadratically with the number of domains.The discriminators remain unchanged and already scale linearly when each domain has its own discriminator.
- Training: Each training iteration uniformly samples two distinct domains, performs the CycleGAN procedure for that pair, and repeats the pass symmetrically in the reverse direction.This strategy eventually covers training across all domain pairs while preserving the underlying CycleGAN losses.
- Training: The training schedule keeps resource growth linear by matching each domain’s expected selection frequency to the two-domain CycleGAN case.The paper reports satisfactory practice using k_2^2/n iterations per domain, although the supplied passage presents the surrounding derivation fragmentarily.
- Relation with CycleGAN: For two domains, ComboGAN becomes exactly equivalent to CycleGAN; with more domains, it assumes encoders produce a shared representation usable by all decoders.This shared-space requirement prevents encoders from specializing to only one decoder.
3. Experiments and Results
Experiments evaluate ComboGAN on four seasonal domains in Alps photographs and fourteen artist domains in paintings. The results show comparable visual quality to CycleGAN on seasons while using fewer networks and shorter training, and demonstrate translations across painters.
- Datasets: The experiments use approximately 6,000 Alps photographs categorized into Spring, Summer, Autumn, and Winter.
- Datasets: The paintings dataset contains approximately 10,000 artworks from 14 different artists.
- Alps seasons: ComboGAN reasonably converts among the four Alps seasons, including semantically meaningful changes such as cloud inversion.
- Alps seasons: ComboGAN maintains comparable quality to CycleGAN while training four networks for 400 epochs instead of twelve networks for 1,200 epochs.
- Fourteen painters: For fourteen painters, translations of one real image from each artist to every other artist exhibit common textures and artist-specific color palettes.
- Fourteen painters: The painter translations include a Corot image rendered as a monochrome pencil/charcoal sketch, coinciding with unfinished sketch examples in the dataset.
4. Conclusion
The conclusion presents ComboGAN as a scalable construction of CycleGAN that retains visual richness beyond two domains. It also identifies extensions and modifications that were deliberately omitted to isolate the scaling effect.
- ComboGAN addresses CycleGAN’s θ(n^2) scaling issue while maintaining visual richness without restricting translation to two domains.
- Additional domains can theoretically be appended by training a new encoder/decoder pair alongside a pretrained model.
- The framework can be extended to UNIT, encoder-decoder layer sharing, and latent-space losses, but these modifications were omitted to isolate scaling effects.
A. Network Architectures
The appendix specifies the network architecture used in the translation experiments through layer specifications for the generator and discriminator.
- The architecture description uses N for neurons, K for kernel size, S for stride size, DCONV for transposed convolution, and RESBLK for residual basic block.
- Table 1 details layer specifications for the Generator, comprising an Encoder and Decoder, and the Discriminator.