Source-linked AI summary
Dual Contrastive Learning for Unsupervised Image-to-Image Translation
Junlin Han, Mehrdad Shoeiby, Lars Petersson, Mohammad Ali Armin
TL;DR
Unpaired image-to-image translation is difficult because adversarial training is under-constrained and cycle-consistency can restrict geometry and diversity. DCLGAN uses dual mappings with separate domain embeddings and contrastive patch correspondence, while SimDCL adds a similarity loss to address mode collapse. The paper reports superior performance across challenging tasks and substantially narrows the gap between unsupervised and supervised methods.
Problem
Unpaired adversarial translation admits multiple mappings, while cycle-consistency can restrict geometry changes and compromise image diversity.
Method
DCLGAN learns dual mappings with separate encoders and embeddings, maximizing mutual information between corresponding input and output patches; SimDCL adds similarity loss for mode collapse.
Results
The method shows quantitative and qualitative superiority across multiple challenging tasks and substantially closes the gap between unsupervised and supervised methods.
Takeaways & Limitations
Dual contrastive learning leverages separate domain representations for unpaired translation, while SimDCL mitigates mode collapse.
Takeaways & Limitations
Cycle-consistency remains better avoided when strict limitations on geometry are not crucial, because adding it worsened FID scores on Cat →Dog and Dog →Cat.
Abstract
from arXiv · showhide
Unsupervised image-to-image translation tasks aim to find a mapping between a source domain X and a target domain Y from unpaired training data. Contrastive learning for Unpaired image-to-image Translation (CUT) yields state-of-the-art results in modeling unsupervised image-to-image translation by maximizing mutual information between input and output patches using only one encoder for both domains. In this paper, we propose a novel method based on contrastive learning and a dual learning setting (exploiting two encoders) to infer an efficient mapping between unpaired data. Additionally, while CUT suffers from mode collapse, a variant of our method efficiently addresses this issue. We further demonstrate the advantage of our approach through extensive ablation studies demonstrating superior performance comparing to recent approaches in multiple challenging image translation tasks. Lastly, we demonstrate that the gap between unsupervised methods and supervised methods can be efficiently closed.
1. Introduction
Unpaired image translation is under-constrained, while cycle-consistency restricts geometry and diversity. DCLGAN addresses these issues with dual contrastive learning, and SimDCL further mitigates mode collapse.
- Unpaired adversarial training admits multiple mappings, making optimization unstable and translation unsuccessful.
- Cycle-consistency constrains geometry changes and forces a bijective relationship that can compromise image diversity.
- CUT maximizes mutual information between corresponding input and output patches with a patch-based, multi-layer PatchNCE loss.
- DCLGAN uses separate domain embeddings and dual learning to maximize patch correspondence while avoiding cycle-consistency.
- SimDCL significantly mitigates mode collapse, while experiments report quantitative and qualitative superiority across multiple image-translation tasks.
2. Related Work
Related work spans supervised paired translation and unsupervised methods based on shared latent spaces or cycle-consistency. Contrastive learning and cycle-breaking approaches motivate DCLGAN’s two-sided unpaired translation framework.
- Supervised methods: Paired supervised translation requires corresponding images from domains X and Y, limiting usage when paired data is unavailable.
- Unsupervised methods: UNIT assumes corresponding images in different domains share a latent representation, while later work extends this framework toward multimodal translation.
- Break the cycle: CycleGAN learns forward and reverse mappings with reconstruction fidelity, but this cycle-consistency constraint can be too restrictive.
- Break the cycle: CouncilGAN, DistanceGAN, and GCGAN relax cycle-based assumptions using multiple generators, one-way translation, or alternative constraints.
- Contrastive learning: Contrastive learning pulls associated signals together and pushes other samples apart, typically using transformed views produced by data augmentation.
3. Method
DCLGAN learns bidirectional mappings for unpaired image translation using separate domain embeddings and patch-level contrastive objectives. SimDCL adds a similarity loss to address mode collapse, while identity loss preserves input properties.
- Architecture: DCLGAN learns two mappings, G: X →Y and F: Y →X, using two generators and discriminators for the respective domains.The discriminators encourage translated images to belong to the correct target domains.
- Patch-based contrastive learning: PatchNCE maximizes mutual information by matching corresponding input and output patches in both translation directions.The contrastive framework compares each query with its corresponding positive patch and other negatives using cosine similarity and temperature scaling.
- Dual embeddings: Separate encoders and projection heads produce domain-specific embeddings intended to capture variability between distinct domains.Features are extracted from selected encoder layers and projected through two-layer MLP heads.
- Objective: DCLGAN combines adversarial, PatchNCE, and identity losses, with identity loss discouraging unnecessary changes and encouraging color preservation.The objective uses λGAN = 1, λNCE = 2, and λidt = 1.
- Similarity loss: SimDCL adds a similarity loss that encourages generated and real images from the same domain to have similar deep features, addressing mode collapse.The variant uses λSIM = 10, runs slower, and is recommended for Photo →Label, semantic segmentation, and similar tasks.
4. Experiments
The experiments evaluate DCLGAN and baselines across six datasets and nine image-translation tasks. Evaluation uses qualitative and quantitative comparisons, primarily FID, with semantic-segmentation metrics for CityScapes.
- Training setup: Training largely follows CUT, using Hinge GAN loss, Adam optimization, and ResNet-based generators with PatchGAN discriminators.DCLGAN is trained for 400 epochs by default, while SimDCL is trained for 200 epochs unless specified.
- Evaluation scope: Six datasets and nine tasks are used to evaluate DCLGAN and competing unsupervised image-translation methods.The tasks include Horse ↔Zebra, Cat ↔Dog, CityScapes Label →City, Van Gogh →Photo, Label ↔Facade, and Orange →Apple.
- Metrics: FID is the main image-quality metric, where lower values indicate smaller Fréchet distance and more realistic generated images.CityScapes additionally uses a pretrained FCN-8 network and three semantic-segmentation metrics to measure correspondences.
- Baselines: The study compares DCLGAN with CUT, FastCUT, CycleGAN, MUNIT, DRIT, DistanceGAN, SelfDistance, and GCGAN.The comparison includes one-sided methods and methods capable of producing diverse outputs for one input image.
5. Results
Across multiple unpaired image-translation tasks, DCLGAN generally outperforms the compared baselines and supports both geometry and texture changes. SimDCL addresses mode collapse, while DCLGAN performs competitively against supervised methods on CityScapes.
- Comparison of different methods: DCLGAN and SimDCL outperform the baselines on CityScapes, Cat →Dog, and Horse →Zebra using FID scores.DCLGAN omits Similarity loss, whereas SimDCL includes it; both are evaluated across the three tasks.
- Comparison of different methods: The qualitative comparisons include failure cases involving unusual poses, rare backgrounds, partial zebra inputs, and human-structure preservation.These cases include transferring zebra textures onto clouds and translating humans into Yosemite.
- Comparison of different methods: DCLGAN retains superior performance across five additional tasks, including Zebra →Horse, Van Gogh →Photo, Dog →Cat, Label →Facade, and Orange →Dog.The comparison reports stronger performance than other methods across various tasks.
- Comparison of different methods: DCLGAN performs both geometry and texture changes, whereas cycle-consistency methods usually fail to perform geometric changes.The contrast is explicitly observed across the additional translation tasks, including Dog →Cat.
- Comparison to supervised methods: DCLGAN performs best in pixACC on CityScapes and is on par with supervised methods on average.The supervised comparison includes Pix2Pix, CRN, and DRPAN using the FCN score.
- Addressing mode collapse via similarity loss: SimDCL is more robust to mode collapse than CUT and DCLGAN on Facade →Label, producing reasonable outputs for different inputs.CUT and DCLGAN generate nearly identical outputs regardless of input in this stress test, while SimDCL captures more facade-label correspondence than CycleGAN.
6. Ablation study
The ablation study tests DCLGAN’s design choices across three translation tasks, finding that separate embeddings, omitted RGB pixels, and the dual setting support performance, while cycle-consistency can hurt geometry changes.
- DCLGAN’s ablation study evaluates individual contributions on Horse →Zebra, Zebra →Horse, and CityScapes.
- Removing the first RGB pixels from PatchNCE improves results, whereas restoring them deteriorates performance on all three tasks.RGB pixels represent the smallest 1 × 1 patches and can provide misleading information.
- Using external negatives gives better quantitative results on Horse →Zebra and close CityScapes results, but lower visual quality than default DCLGAN.The external-negative variant provides 255 internal and 256 external negatives per query, compared with 255 internal negatives by default.
- Sharing one encoder and MLP across domains deteriorates results because one embedding fails to capture variability between distinct domains.
- Adding cycle-consistency produces no improvements and yields worse FID scores of 71.1 for Cat →Dog and 35.5 for Dog →Cat.The authors conclude that cycle-consistency is better avoided when strict geometry limitations are not crucial.
- Removing the dual setting worsens results, indicating that dual settings learn better domain embeddings and stabilize training.
7. Conclusion
The conclusion presents dual contrastive learning as a stronger way to exploit contrastive learning for unpaired translation. It also highlights SimDCL’s mitigation of mode collapse and the method’s ability to narrow the supervised–unsupervised gap.
- DCLGAN uses a dual setting to better leverage contrastive learning for unsupervised unpaired image-to-image translation.
- The SimDCL variant mitigates mode collapse, while the overall method substantially closes the gap between unsupervised and supervised methods on challenging datasets such as CityScapes.
- The method uses ResNet generators with nine residual blocks and PatchNCE features from four encoder layers.The selected patch sizes are 9×9, 15×15, 35×35, and 99×99, with 256 sampled locations per layer.
- DCLGAN employs PatchGAN discriminators that evaluate local 70x70 image patches for domain realism.
- SimDCL uses four light projection networks that map 256-dimensional features to 64-dimensional vectors.
A.1.4 Additional training details
Additional training details specify optimization, initialization, and image preprocessing for SimDCL and DCLGAN.
- SimDCL uses Adam with β1 = 0.5 and β2 = 0.999, updating all projection networks with learning rate 0.0002.
- Both models use Xavier initialization, train on random 256x256 crops from 286x286 images, and test on 256x256 images.
A.1.5 Additional evaluation details
The paper evaluates image quality with FID and semantic correspondence on CityScapes with FCN scores, using official implementations and pretrained evaluation networks.
- FID is computed with the official PyTorch implementation and default CUT evaluation settings.
- FCN score measures label–image correspondence on CityScapes using a pretrained FCN-8 network applied to generated photos.
A.2. Qualitative results of ablations
Qualitative ablations indicate that DCLGAN produces more realistic images than its variants, with each proposed contribution contributing to efficiency.
- DCLGAN generates more realistic images than the evaluated ablation variants.The ablations remove or modify RGB pixels, external negatives, encoder sharing, cycle-consistency, and the dual setting.
A.3. Additional Results
Additional qualitative results cover multiple translation tasks and show strong performance, including geometry and texture changes, while revealing dataset and foreground-background limitations.
- DCLGAN is evaluated qualitatively across nine tasks, with additional comparisons shown for most tasks except Facade →Label.
- Horse ↔Zebra inference is imperfect because the dataset contains horse images collected from ImageNet using the keyword “wild horse.”
- DCLGAN sometimes fails to distinguish foreground and background, similarly to most recent methods.
- DCLGAN performs best on Cat ↔Dog geometry changes and generates realistic outputs with reasonable structure.CycleGAN fails to perform geometry changes on this task.
- DCLGAN consistently outperforms other methods on texture changes, while CUT fails to modify the color of whole oranges in one comparison.
- Similarity loss acts as a regularization term that prevents mode collapse when outputs risk losing diversity and realism.
- External negatives may improve quantitative results on certain tasks but usually worsen qualitative results, including merged pedestrians and cars.