Source-linked AI summary
Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
Amjad Almahairi, Sai Rajeswar, Alessandro Sordoni, Philip Bachman, Aaron Courville
TL;DR
Unpaired domain translation is difficult when the true relationship is many-to-many, because CycleGAN assumes deterministic one-to-one mappings. The paper introduces Augmented CycleGAN, which adds latent variables and learns mappings in augmented spaces; experiments report diverse outputs and effectiveness across image translation tasks, including semi-supervised face-attribute mapping.
Problem
CycleGAN assumes deterministic one-to-one inter-domain mappings, limiting tasks that require flexible many-to-many relationships.
Method
Augmented CycleGAN augments each domain with auxiliary latent variables and extends cycle-consistent training to the resulting spaces.
Results
The model produces diverse outputs, maps across substantially different domains, and achieves competitive results for semi-supervised face-attribute mapping.
Takeaways & Limitations
Augmented CycleGAN provides stochastic many-to-many cross-domain mappings and is effective across the evaluated image translation tasks.
Takeaways & Limitations
CycleGAN’s deterministic mappings can learn arbitrary one-to-one relationships instead of faithfully representing complex structured conditional distributions.
Abstract
from arXiv · showhide
Learning inter-domain mappings from unpaired data can improve performance in structured prediction tasks, such as image segmentation, by reducing the need for paired data. CycleGAN was recently proposed for this problem, but critically assumes the underlying inter-domain mapping is approximately deterministic and one-to-one. This assumption renders the model ineffective for tasks requiring flexible, many-to-many mappings. We propose a new model, called Augmented CycleGAN, which learns many-to-many mappings between domains. We examine Augmented CycleGAN qualitatively and quantitatively on several image datasets.
1. Introduction
Unpaired domain mapping is useful when paired data are scarce, but CycleGAN’s one-to-one assumption cannot represent diverse or substantially different cross-domain relationships. Augmented CycleGAN addresses this by adding latent variables and learning many-to-many mappings after marginalization.
- Motivation: Paired data can be scarce or difficult to obtain, motivating unsupervised domain mapping for tasks such as face transfiguration.An unsupervised model may also help when only relatively few paired examples are available.
- CycleGAN limitation: CycleGAN reduces possible mappings through cycle-consistency, requiring source-to-target-to-source translation to recover the starting image.This constraint creates a strong connection across domains.
- CycleGAN limitation: CycleGAN learns one output per input, whereas shoe silhouettes may correspond to multiple shoe styles and substantially different domains may have one-to-many relationships.The introduction gives descriptive facial attributes-to-face images as an example of a difficult domain gap.
- Proposed approach: Augmented CycleGAN augments each domain with auxiliary latent variables and extends CycleGAN’s training procedure to the resulting spaces.The mappings take a source sample and latent variable as input and produce a target sample and latent variable.
- Proposed approach: The model is one-to-one in augmented spaces but many-to-many in the original domains after marginalizing the latent variables.The paper reports diverse outputs, mappings across substantially different domains, and competitive semi-supervised face-attribute results.
2. Unsupervised Learning of Mappings Between Domains
The paper formulates unpaired domain translation as conditional modeling and explains why CycleGAN’s marginal matching and cycle-consistency favor deterministic inverses. It then considers stochastic mappings and motivates augmented spaces as an alternative for expressive cross-domain relationships.
- Problem formulation: The objective is to recover potentially many-to-many mappings between domains from unpaired samples by estimating both conditional distributions from domain marginals.The formulation assumes elements in the two domains are highly dependent.
- CycleGAN model: CycleGAN uses mappings in both directions, with marginal matching and cycle-consistency as its main constraints.Marginal matching aligns generated outputs with target distributions, while cycle-consistency reconstructs the original input.
- CycleGAN model: CycleGAN’s adversarial losses enforce marginal matching, while reconstruction-based cycle-consistency keeps translated samples close to their sources.The two losses are combined through a hyper-parameter balancing marginal matching and cycle-consistency.
- Limitations of CycleGAN: CycleGAN deterministically models each conditional and therefore learns arbitrary one-to-one inverse mappings instead of faithfully capturing complex structured conditional distributions.This is especially problematic when domain complexities differ and one direction is generally one-to-many.
- Stochastic CycleGAN: Stochastic CycleGAN adds Gaussian latent noise to both mappings, allowing different sampled latent values to produce multiple outputs for one input in principle.Its mappings take a source sample and auxiliary noise vector as input.
- Stochastic CycleGAN: Stochastic CycleGAN’s cycle-consistency can make the mappings ignore latent noise, forcing many-to-one behavior that collapses toward one-to-one mutual inverses.The paper notes that a VAE-like encoder and variational bound could potentially mitigate this degeneracy.
3. Approach
Augmented CycleGAN extends CycleGAN to augmented domain spaces with latent variables, enabling many-to-many mappings while retaining marginal matching and cycle consistency. The model uses generators, encoders, adversarial losses, and augmented-space cycles, with paired data optionally supporting semi-supervised training.
- Augmented CycleGAN: Augmented CycleGAN maps pairs from A × Zb to B × Za and vice versa, using latent spaces to capture information missing during domain transformations.Latent variables enable stochastic outputs and preserve information needed for reconstruction.
- Model components: The model uses conditional generators GAB and GBA plus encoders EA and EB to produce domain samples and infer latent codes.The mappings operate on augmented spaces, with standard Gaussian priors for the latent variables.
- Stochastic mappings: Sampling different zb values generates multiple outputs for the same a, while inferred za retains information about a that generated b does not capture.The reverse direction applies the analogous construction for samples from B.
- Training objectives: Training optimizes marginal matching and cycle-consistency losses over augmented spaces, with separate objectives for both mapping directions.Adversarial matching also regularizes the marginalized latent-code distributions toward simple priors.
- Cycle consistency: Two augmented-space cycle constraints reconstruct both the original domain sample and the associated latent code after forward and reverse mappings.The encoder recovers information lost from the generated domain sample, while latent-code reconstruction increases dependence between zb and b conditioned on a.
- Semi-supervised learning: Paired samples can provide supervision for the mappings in a semi-supervised setting, with encoders inferring latent codes associated with observed pairs.The supervised mapping cost is supplemented by adversarial encoder regularization.
- Design choices: Injecting latent codes near network inputs was important for allowing them to capture diversity in generated samples.Earlier injection gives the codes more subsequent layers through which to represent high-level output variation.
4. Related Work
Related work on unpaired cross-domain mappings commonly uses GANs and cycle-consistency, while other approaches use shared latent spaces or powerful sequence decoders. Augmented CycleGAN instead uses auxiliary latent variables for multimodal mappings and addresses settings involving substantially different domains.
- Unpaired image translation: Recent image-to-image translation methods largely rely on GANs and cycle-consistency constraints to learn and restrict unpaired cross-domain mappings.These approaches aim to produce compelling images while constraining the space of possible mappings.
- Alternative formulations: Shared-latent-space approaches can constrain the learnable mappings when the domains differ substantially, such as class labels and images.The cited related work contrasts this limitation with the proposed use of auxiliary latent variables.
- Language translation: Unsupervised language translation and text style transfer also use cycle-consistency, relying heavily on RNN-based decoders to capture complex cross-domain relationships.Augmented CycleGAN proposes auxiliary latent variables instead.
- Multimodal mappings: BiCycleGAN learns multimodal mappings in a fully supervised setting, whereas Augmented CycleGAN focuses on learning many-to-many mappings from unpaired data.Both models use stochastic mappings and encoders for multimodal targets, but their supervision settings differ.
5. Experiments
Experiments evaluate Augmented CycleGAN on edges-to-shoes, male-to-female faces, and attributes-to-faces translation. The results show diverse conditional generation, robustness to edge corruption, and competitive semi-supervised attribute prediction.
- 5.1. Edges-to-Photos: AugCGAN outperforms all listed baselines on shoe reconstruction from test-set edges, including semi-supervised Δ-GAN, while 10% supervision improves its predictions further.The evaluation estimates latent codes by optimization and averages reconstruction errors over 200 test samples.
- 5.1. Edges-to-Photos: On edges-to-shoes translation, AugCGAN and StochCGAN generate diverse shoes and reconstruct edges and shoes when conditioned on real edges.The edges are sampled from the data distribution, with diversity driven largely by the latent code z_b.
- 5.1. Edges-to-Photos: When edges are generated by the model, StochCGAN collapses to one shoe across latent codes, whereas AugCGAN captures conditional diversity and maintains diverse samples across cycles.The real-versus-generated edge distinction exposes a regime-switching problem for StochCGAN; repeated cycles show the same contrast.
- 5.1. Edges-to-Photos: AugCGAN remains more robust to corrupted generated edges than StochCGAN, indicating that information is stored in latent codes rather than hidden entirely in edges.The comparison uses L1 shoe-reconstruction error as Gaussian noise injected into generated edges increases.
- 5.2. Male-to-Female: In male-to-female translation, AugCGAN achieves LPIPS diversity 0.108 ± 0.003, compared with 0.059 ± 0.001 for low-level latent injection and 0.008 ± 0.000 for StochCGAN.The results highlight the importance of processing latent codes through multiple layers and optimizing cycle-consistency appropriately.
- 5.3. Attributes-to-Faces: In the semi-supervised Attributes-to-Faces task, the model is evaluated with Precision and normalized Discounted Cumulative Gain, while generated faces are reasonably diverse for fixed attributes.Training uses all unpaired data plus a small amount of paired data; Table 3 reports supervision levels of 1% and 10%.
6. Conclusion
Augmented CycleGAN learns many-to-many cross-domain mappings from unpaired data by using auxiliary noise to capture multimodal conditionals. Experiments support its effectiveness in image translation and semi-supervised face–attribute mapping.
- Augmented CycleGAN learns many-to-many cross-domain mappings from unpaired data.
- Auxiliary noise enables the model to capture multimodal conditionals through stochastic mappings.
- Quantitative and qualitative experiments verify the approach's effectiveness in image translation tasks.
- The model is applied effectively to semi-supervised learning for mapping between attributes and faces.