Source-linked AI summary
Unpaired Image-to-Image Translation using Adversarial Consistency Loss
Yihao Zhao, Ruihai Wu, Hao Dong
TL;DR
Unpaired image-to-image translation commonly relies on cycle consistency, whose strict pixel-level constraint can preserve unwanted features and limit geometric changes. ACL-GAN introduces adversarial-consistency loss to retain important source features without reconstructing a specific source image, achieving state-of-the-art results across three challenging tasks.
Problem
Cycle consistency can preserve unwanted objects and textures and constrain geometric changes because translated images must retain enough information to reconstruct the input.
Method
ACL-GAN combines adversarial-consistency loss with adversarial and additional losses to preserve shared source-target features without requiring pixel-level reconstruction of a specific source image.
Results
ACL-GAN achieves state-of-the-art results on glasses removal, male-to-female translation, and selfie-to-anime translation.
Takeaways & Limitations
ACL-GAN can maintain source-target commonalities while performing shape modifications and removing large objects without unrealistic traces.
Takeaways & Limitations
The method does not yet handle datasets with complex backgrounds well, and synchronous translation between domains decreased performance.
Abstract
from arXiv · showhide
Unpaired image-to-image translation is a class of vision problems whose goal is to find the mapping between different image domains using unpaired training data. Cycle-consistency loss is a widely used constraint for such problems. However, due to the strict pixel-level constraint, it cannot perform geometric changes, remove large objects, or ignore irrelevant texture. In this paper, we propose a novel adversarial-consistency loss for image-to-image translation. This loss does not require the translated image to be translated back to be a specific source image but can encourage the translated images to retain important features of the source images and overcome the drawbacks of cycle-consistency loss noted above. Our method achieves state-of-the-art results on three challenging tasks: glasses removal, male-to-female translation, and selfie-to-anime translation.
1 Introduction
ACL-GAN replaces strict cycle consistency with adversarial consistency to preserve shared source-target features while allowing shape changes and object removal. It achieves state-of-the-art results on glasses removal, male-to-female translation, and selfie-to-anime translation.
- The method is designed to bypass unnecessary features while maintaining commonalities between source and target domains.
- Cycle consistency preserves input information for reconstruction, but can retain unwanted objects and textures and constrain shape changes.
- Adversarial-consistency loss encourages shared source-target features distributionally without requiring pixel-level reconstruction of a specific source image.
- ACL-GAN achieves state-of-the-art results on glasses removal, male-to-female translation, and selfie-to-anime translation.
2 Related Work
Related work develops image translation for applications including colourisation, super-resolution, and inpainting, while unpaired methods address settings where paired data are unavailable. Existing approaches preserve source properties through pixel-level constraints or cycle consistency, and CouncilGAN replaces cycle consistency with multiple generator-discriminator pairs.
- Image-to-image translation supports applications such as colourisation, super-resolution, and image inpainting, while Pix2Pix requires paired images for supervision.
- Unpaired image-to-image translation targets applications where paired data are unavailable or difficult to collect.
- Prior methods preserve source properties using pixel gradients, pixel values, pairwise sample distances, or shared latent-space and cycle-consistency assumptions.
- CouncilGAN replaces cycle consistency with duplicate generators and discriminators whose council loss encourages translated images to retain source information.
3 Method
ACL-GAN replaces strict pixel-level cycle consistency with adversarial consistency that preserves source features at the distribution level, while combining translation, identity, and mask losses. Its consistency discriminator and multi-modal outputs allow translated-back images to vary around the source rather than reproduce one exact image.
- Model architecture: Two generators map image-noise pairs into the source and target domains, while domain discriminators distinguish real from generated images and a consistency discriminator enforces source-feature preservation.The generators use image encoders, noise encoders, and decoders; the noise vector enables diverse outputs.
- Overall objective: ACL-GAN uses adversarial-translation, adversarial-consistency, identity, and bounded-focus-mask losses to translate images while preserving important features and background.The adversarial-translation loss matches target-domain distributions; adversarial consistency preserves source features; identity and mask losses improve quality and background preservation.
- Adversarial-Consistency Loss: Adversarial consistency matches the joint distributions of source-reference pairs and translated-back pairs instead of requiring a translated image to equal one specific source image.Using the source image as a reference lets the consistency discriminator operate at the feature level rather than impose pixel-level identity.
- Adversarial-Consistency Loss: Multi-modal noise is essential because it lets translated-back images occupy multiple plausible points around the source, avoiding the strong constraint that otherwise resembles cycle consistency.With only one output per source image, matching the paired distributions would nearly require exact equality.
- Other Losses: Identity loss encourages approximate identity mappings on target-domain inputs, improving feature preservation and image quality while stabilising training and avoiding mode collapse.It also ensures the source image lies within the distribution of multi-modal reconstructions.
- Other Losses: The bounded focus mask forms each output by blending generated content with the source image, restricting changes while preserving the background.Mask-size constraints encourage sufficient foreground changes, prevent identical outputs under different noise vectors, and push mask values toward binary segmentation.
4 Experiments
ACL-GAN is evaluated through ablations and comparisons on glasses removal, male-to-female translation, and selfie-to-anime translation using unpaired data. Across these experiments, the full model outperforms ablated settings and baselines on reported quantitative metrics while producing more suitable qualitative translations.
- Ablation Studies: The full ACL-GAN with total loss outperforms ablated settings on male-to-female translation for both FID and KID.The ablations remove adversarial-consistency, identity, or mask losses.
- Ablation Studies: Without adversarial-consistency loss, facial features such as skin colour, wrinkles, and teeth are harder to preserve, and quantitative results are worst.This comparison uses ACL-A, which retains identity loss and bounded focus mask.
- Ablation Studies: The bounded focus mask improves perceptual quality and image quality by directing the generator toward essential translation regions and better maintaining backgrounds.ACL-GAN with the mask is qualitatively better than ACL-M without it.
- Comparison with Baselines: For glasses removal, ACL-GAN leaves fewer glasses traces and outperforms every baseline on both FID and KID, including MUNIT and CouncilGAN.The comparison includes CycleGAN, MUNIT, Fixed-Point GAN, DRIT++, and CouncilGAN.
- Comparison with Baselines: ACL-GAN outperforms all baselines on glasses removal, male-to-female translation, and selfie-to-anime translation according to FID and KID results.Table 3 reports lower scores as better and includes U-GAT-IT in light mode.
- Comparison with Baselines: For male-to-female translation, ACL-GAN produces diverse outputs from different noise vectors and more feminine faces while preserving important image features better than several baselines.Compared with cycle-consistent methods, outputs have no beard, longer hair, and more feminine lips and eyes; MUNIT preserves hue less effectively.
- Comparison with Baselines: For selfie-to-anime translation, ACL-GAN produces images more consistent with anime style while preserving source features such as haircut and face rotation.It outperforms light-mode U-GAT-IT despite using fewer parameters.
5 Limitations and Discussion
ACL-GAN maintains source–target commonalities while allowing shape modifications and large-object removal without unrealistic traces. Despite outperforming state-of-the-art methods on three tasks, it has failure cases and struggles with complex backgrounds.
- ACL-GAN maintains commonalities between source and target domains while supporting shape modifications and large-object removal without unrealistic traces.
- The method outperforms state-of-the-art methods quantitatively and qualitatively on three challenging tasks, although typical failure cases remain.
- ACL-GAN does not handle datasets with complex backgrounds well, making complex-background translation a direction for future work.