Source-linked AI summary
Unsupervised Attention-guided Image to Image Translation
Youssef A. Mejjati, Christian Richardt, James Tompkin, Darren Cosker, Kwang In Kim
TL;DR
Unsupervised image-to-image translation struggles to focus on individual objects without changing backgrounds or object interactions. The paper jointly trains attention mechanisms with generators and discriminators to select relevant regions, achieving more realistic mappings without supervision. The approach performs especially well with multi-scale objects and varying backgrounds, while remaining limited for outlier appearances and substantial shape changes.
Problem
Unsupervised image-to-image translation struggles to change individual scene objects while preserving irrelevant background content and object interactions.
Method
The method jointly trains attention networks with generators and discriminators, using continuous per-pixel attention maps to blend translated foregrounds with input backgrounds.
Results
The approach learns relevant regions without supervision and obtains more realistic translations, with the lowest KID score across all evaluated mappings.
Takeaways & Limitations
Learned attention maps can both improve unsupervised translation quality and indicate where the discriminator focuses when judging images as real or fake.
Takeaways & Limitations
The approach is not robust to outlier object appearances and shape changes between domains that require altering regions outside the attended areas.
Abstract
from arXiv · showhide
Current unsupervised image-to-image translation techniques struggle to focus their attention on individual objects without altering the background or the way multiple objects interact within a scene. Motivated by the important role of attention in human perception, we tackle this limitation by introducing unsupervised attention mechanisms that are jointly adversarialy trained with the generators and discriminators. We demonstrate qualitatively and quantitatively that our approach is able to attend to relevant regions in the image without requiring supervision, and that by doing so it achieves more realistic mappings compared to recent approaches.
1 Introduction
Image-to-image translation maps images between domains, but unsupervised methods must infer which scene regions to translate without paired or aligned data. Existing approaches can alter irrelevant background content because they lack an explicit attention mechanism.
- Image-to-image translation maps an image from a source domain to a target domain across applications including colorization, super-resolution, style transfer, adaptation, and augmentation.
- Paired or aligned source–target data restricts translation applications, especially when corresponding images may not exist.
- Unsupervised methods such as DiscoGAN and CycleGAN use cyclic losses to encourage faithful reconstruction after mapping translated images back to the original domain.
- Existing approaches struggle to translate only relevant scene objects, such as changing horses to zebras while preserving the rest of the image.
- Without explicit attention, networks typically minimize divergence between whole-image source and target distributions, which can affect irrelevant regions.
2 Related work
Prior work addresses unpaired translation, localized attention, saliency, and feature selection, but these approaches impose assumptions or practical constraints. The paper positions input-image attention as a way to preserve backgrounds while translating foreground objects across scales.
- Image-to-image translation: Pix2pix uses a source-conditioned GAN with an L1 loss against ground-truth maps, requiring paired images from the source and target domains.
- Image-to-image translation: CoGAN and UNIT address unsupervised translation by sharing parameters or assuming a shared low-dimensional latent space between source and target domains.
- Image-to-image translation: Existing methods can translate foreground objects at multiple scales but often alter backgrounds in unwanted ways, producing unrealistic translations.
- Attention learning: Localized square attention maps assume equal object sizes and require tuning the number and size of regions.
- Attention learning: Supervised saliency approaches require ground-truth masks or other supervision and therefore cannot be directly applied to this unsupervised translation setting.
- Attention learning: Unsupervised recurrent visual attention uses learned square regions and reinforcement learning, while feature-space attention selects task-relevant activations rather than input-image regions.
- Attention learning: Self-attention GANs model non-local feature relationships, and recursive generation decouples foreground and background generation sequentially.
3 Our approach
The approach augments CycleGAN-style translation with jointly trained attention networks that locate regions to translate while preserving the background. Attention-guided generation, cycle consistency, and staged discriminator training address object localization, content preservation, and training instability.
- Attention-guided generator: Two attention networks select per-pixel regions to translate, and the masked generated foreground is combined with the inverse-masked source background.AS and AT are trained in tandem with the generators.
- Attention-guided generator: The mapped image s′ combines the attention-masked generator output with the inverse-masked input image, preserving unattended regions.This is the foreground-plus-background construction in Equation 1.
- Attention-guided generator: Attention maps are continuous mattes in [0,1], enabling differentiable training, uncertainty during convergence, and smoother object boundaries.The matte representation also helps the network compose edges without fringing artifacts.
- Loss function: Cycle consistency combines with adversarial training to preserve attended content, encourage sharper attention maps, and require attention predictions to remain consistent across mappings.The inverse mapping uses the attention map extracted from the translated image.
- Attention-guided discriminator: Whole-image discrimination creates a foreground–background distribution tension, causing background painting and progressively expanding attention masks.The attention map can converge toward a fully attended image.
- Attention-guided discriminator: The discriminator is first trained on full images for 30 epochs, then switched to attended regions after the attention networks develop to reduce mode collapse risk.Real samples depending on initially untrained masks can otherwise destabilize joint GAN training.
4 Experiments
Experiments compare the attention-guided method with established translation baselines across multi-scale object datasets, qualitative examples, KID scores, and ablations. The method generally preserves irrelevant backgrounds and achieves the lowest KID, while attention and cycle-consistency choices materially affect performance.
- Datasets: The experiments use Apple↔Orange, Horse↔Zebra, and Lion↔Tiger datasets containing objects at different scales and backgrounds.The Lion↔Tiger mapping does not require the attention-guided discriminator component.
- Qualitative results: The method learns relevant regions, ignores backgrounds, leaves images unedited when the source object is absent, and avoids hallucinated target textures.Holistic methods instead alter irrelevant backgrounds and hallucinate target-object patterns in source-absent cases.
- Qualitative results: Competing methods show distinct failures: DiscoGAN mixes foreground and background, DualGAN still alters backgrounds, CycleGAN changes background elements, and UNIT can produce unrealistic images.Residual attention modules do not improve generated-image quality, particularly when relevant source objects are absent.
- Limitations: The approach fails on an albino tiger, whose outlier appearance prevents foreground identification and causes background changes.This is a challenging within-domain appearance case rather than the typical multi-scale, varied-background setting.
- Quantitative results: The approach achieves the lowest KID score in all mappings, with CycleGAN the next best-performing approach.UNIT is second-lowest, DualGAN follows closely, RA is worse than regular CycleGAN, and DiscoGAN performs worst by mean KID.
- Ablation study: Removing cycle consistency causes very high KID and an onset of mode collapse, while computing attention on both mapping directions performs worse by indirectly enforcing attention-map similarity.Single-attention variants perform best after the final implementation, and replacing the attention-guided discriminator raises mean KID values.
5 Conclusion
The paper concludes that unsupervised attention improves image-to-image translation by restricting changes to relevant regions and using the discriminator to learn attention maps without supervision. However, the approach remains limited for shape changes between domains because such changes may require modifying areas outside attended regions.
- Conclusion: Recent unsupervised methods also translate irrelevant regions, whereas the proposed attention mechanism improves generated-image quality by better separating foreground and background.The method requires no additional supervision and learns attention maps jointly with adversarial translation.
- Conclusion: The learned attention maps may reflect where the discriminator looks before deciding whether an image is real or fake, making them useful for investigating adversarial networks.This is presented as a supported interpretation of the attention maps’ role.
- Future work: The approach is not robust to shape changes such as translating a horse into a bird because transfer must occur within attended regions.The paper illustrates this limitation with zebra-to-lion translation.
A Network architecture
The architecture uses CycleGAN-like generators and discriminators alongside dedicated attention networks that produce per-pixel masks. Instance normalization, specified convolutional blocks, nearest-neighbor upsampling, and LSGAN are used across the components.
- Generators: The generators follow a CycleGAN-like encoder–decoder architecture with convolutional, residual, transpose-convolutional, and tanh output layers.The specified sequence is c7s1-32-R, c3s2-64-R, c3s2-128-R, nine r128 blocks, tc64s2, tc32s2, and c3s1-3-T.
- Attention networks: The attention networks use instance normalization, nearest-neighbor upsampling, stride-1 convolutions with ReLU activations, and a sigmoid output.Their architecture is c7s1-32-R, c3s2-64-R, r64, up2, c3s1-64-R, up2, c3s1-32-R, c7s1-1-S.
- Discriminators: The discriminators adopt the CycleGAN architecture with instance normalization, Leaky-ReLU activations of slope 0.2, and a five-layer convolutional structure.Instance normalization is removed when discriminators receive masked foregrounds after 30 epochs.
- Adversarial objective: Least Square GAN is used because the authors find that it helps produce sharper images.
B Limitation of our approach
The approach is not robust to significant geometric changes between domains. Shape changes can require modifications outside attended regions, causing attention to spread into the background while cycle consistency limits the transformation.
- Limitation of our approach: The algorithm lacks robustness to significant geometric changes, including Lion →Zebra and Zebra →Lion mappings.
- Limitation of our approach: Shape changes typically require altering image parts outside the attended regions, so the attention maps may cover background areas.
- Limitation of our approach: Cycle-consistency constraints limit these geometric changes even when the attention maps expand into the background.
C Hyper-parameter tuning
Training is divided into two stages, with attention networks trained initially and then frozen while discriminators receive thresholded foregrounds. Threshold selection affects realism, while the staged procedure addresses attention spreading into the background.
- Training stages: The first stage jointly trains generators, attention networks, and discriminators using holistic images.
- Training stages: In the second stage, attention-network training stops and discriminators are trained using foreground-only inputs.
- Mask thresholding: Attention masks are thresholded into binary masks before foregrounds are fed to the discriminator, preventing fractional values from influencing realism learning.
D Additional results
Additional experiments show that attention can cover the entire image for holistic translations, while ablations and qualitative examples examine realism, training components, domain transfers, and irrelevant backgrounds.
- Hyper-parameter effects: Varying the threshold parameter shows that low values give similar results, while higher values yield less realistic mappings.
- Additional results: For Summer →Winter and Winter →Summer mappings, the algorithm produces attention maps focused on the entire image.These mappings use 200 rather than 100 epochs, and Eq. 7 is not applied because foreground and background are not clearly distinguished.
- Additional results: The qualitative evaluation covers translation examples across six datasets and domains without the object of interest.
- Ablation experiments: The final formulation produces sharper and more realistic images in ablation experiments, while removing cycle consistency collapses GAN training.
- Additional results: On images without horses or zebras, attention maps ignore irrelevant backgrounds and reproduce the inputs, whereas existing algorithms hallucinate zebra stripes.