Source-linked AI summary
Attention-GAN for Object Transfiguration in Wild Images
Xinyuan Chen, Chang Xu, Xiaokang Yang, Dacheng Tao
TL;DR
Object transfiguration requires changing selected objects without altering their backgrounds, yet classical generative mappings jointly detect and transform them. Attention-GAN separates those roles with sparse, consistent attention and transformation networks, and experiments report higher-quality, more background-consistent results, with further improvement from segmentation supervision.
Problem
Classical generative networks for object transfiguration jointly detect objects of interest and transform them, risking mistaken transformation of background regions.
Method
Attention-GAN factorizes generation into attention and transformation networks, using sparse and attention-cycle losses plus layered composition of transformed objects with original backgrounds.
Results
The proposed algorithm produces higher-quality object transfiguration with improved background consistency compared with CycleGAN, and segmentation supervision further improves attention accuracy.
Takeaways & Limitations
Investigating attention is necessary for object transfiguration, and separating localization from transformation yields practically preferred generated results.
Abstract
from arXiv · showhide
This paper studies the object transfiguration problem in wild images. The generative network in classical GANs for object transfiguration often undertakes a dual responsibility: to detect the objects of interests and to convert the object from source domain to target domain. In contrast, we decompose the generative network into two separat networks, each of which is only dedicated to one particular sub-task. The attention network predicts spatial attention maps of images, and the transformation network focuses on translating objects. Attention maps produced by attention network are encouraged to be sparse, so that major attention can be paid to objects of interests. No matter before or after object transfiguration, attention maps should remain constant. In addition, learning attention network can receive more instructions, given the available segmentation annotations of images. Experimental results demonstrate the necessity of investigating attention in object transfiguration, and that the proposed algorithm can learn accurate attention to improve quality of generated images.
1 Introduction
Object transfiguration changes selected objects while preserving the background, but conventional generative networks combine object detection and transformation. Attention-GAN separates these roles and uses sparse, cycle-consistent attention, with optional segmentation supervision.
- Object transfiguration converts a particular object type while leaving background regions unchanged.Examples include transforming horses into zebras while preserving grassland and trees.
- Existing one-shot generative networks jointly detect objects of interest and transform them, which can confuse optimization goals.The network may not clearly distinguish improving object detection from improving transfiguration.
- Background regions can be mistakenly included in the transformation, seriously reducing generated-image quality.The paper gives unintended background transformation as a concrete failure mode.
- Attention-GAN factorizes the generative network into an attention network for locating objects and a transformation network for converting them.A sparse attention constraint focuses limited attention energy on priority regions, while layered composition restores the original background.
- The model uses adversarial training with cycle-consistent loss for unpaired data and can additionally learn attention under segmentation supervision.The paper states that supervised attention can improve performance when segmentation results are available.
2 Related Work
Related work covers supervised and unpaired image-to-image translation, conditional GANs, semantic disentanglement, and attention mechanisms. Existing approaches address object transfiguration, but segmentation annotations can be difficult to obtain at scale, especially for testing.
- Image-to-image translation methods include conditional GANs and cycle-consistency-based approaches for unpaired domains.These methods establish connections between source and target domains without paired examples.
- GAN research has extended conditional image generation to tasks such as image-to-image translation and text-to-image generation.Conditional GANs synthesize images using input information such as categories, images, or text.
- Mask-conditional contrast-GAN separates background from semantic object changes using segmentation annotations in training and testing.The cited work addresses object transfiguration by exploiting semantic masks.
- Collecting segmentation masks for many images is difficult, particularly during the test phase.This limits the practicality of approaches requiring annotations for both training and testing.
- Attention mechanisms let models focus on relevant image regions or features instead of compressing an entire image into a static representation.The paper situates attention across vision and language tasks, including classification, captioning, visual question answering, and segmentation.
3 Preliminaries
CycleGAN-style preliminaries use adversarial and cycle-consistency losses to learn mappings between unpaired domains. For object transfiguration, however, the mappings must both locate target objects and transform them, creating a dual responsibility.
- Image-to-image translation learns mappings G: X → Y and F: Y → X using samples from two domains.The discriminators distinguish real target-domain images from translated images, while mappings seek to fool them.
- Adversarial objectives train discriminators to distinguish real and translated images while mapping functions generate outputs that fool the discriminators.This is formulated as a min-max game between mapping functions and discriminators.
- Cycle consistency penalizes discrepancies when translated images are mapped back to the original image space.It is introduced because a sufficiently expressive network could otherwise map inputs to arbitrary permutations of target images.
- Combining adversarial and cycle-consistency losses supports one-to-one correspondence mappings and cross-domain relation discovery.The full objective balances these two objectives with a relative-weight parameter.
- In object transfiguration, the generative mappings must detect objects of interest and transfigure them, which can confuse the generative network’s aims.The background may be encoded in arbitrary representations as long as it can be reconstructed, without guaranteeing background preservation.
4 Model
Attention-GAN separates object localization, object transformation, and discrimination, then combines transformed objects with original backgrounds. Attention-cycle and sparse losses preserve region consistency and focus, while segmentation labels provide an optional supervised route.
- The model has attention, transformation, and discriminative networks that respectively locate regions of interest, transform objects, and distinguish real from generated images.The final image combines the transformed object with the original background through a layered operator.
- 4.1 Formulations: A layered operation combines the attention-weighted transformed region with the original image background to construct the final image.This design keeps background content outside the selected transformed region.
- 4.1 Formulations: The attention network outputs a spatial score map with values from 0 to 1, assigning higher scores to objects and suppressing background.The transformation branch produces a target-domain-like image before layered composition with the input background.
- 4.2 Attention Losses: Attention cycle consistency requires attention maps before and after transfiguration to identify the same regions of interest.For example, the horse region and its transformed zebra region should remain spatially consistent.
- 4.2 Attention Losses: Sparse loss encourages attention to concentrate on a small object-related region rather than the whole image.The full objective combines adversarial, cycle-consistency, attention-cycle, and sparse terms with balancing weights.
- 4.3 Supervised Attention: Segmentation annotations can supervise predicted attention maps by minimizing their discrepancy from binary segmentation labels.When semantic annotations are used, the supervised objective replaces the attention-cycle and sparse constraints.
5 Experiments
The experiments evaluate Attention-GAN on horse↔zebra, tiger↔leopard, and apple↔orange transfiguration tasks, using unsupervised and supervised settings alongside comparisons with CycleGAN.
- Evaluation design: The evaluation compares unsupervised Attention-GAN with a state-of-the-art unpaired translation method and studies attention sparse loss and supervised attention.The experiments also include variant comparisons and supervised Attention-GAN results.
- Datasets: Experiments cover horse↔zebra, tiger↔leopard, and apple↔orange object transfiguration tasks.A supervised experiment additionally evaluates horse↔zebra using MSCOCO images and annotations.
- Training strategy: Training uses an initial learning rate of 0.0002 for 100 epochs, followed by linear decay to zero over the next 100 epochs.All experiments use Adam with batch size 1 and previously generated images for discriminator updates.
- Qualitative results: Figure 3 presents original images, synthesized images, and predicted attention maps for horse↔zebra, leopard↔tiger, and apple↔orange tasks.The figure is intended to show manipulation of attention regions while preserving background consistency.
5.1 Assessment of Image Quality
Image quality is assessed by measuring how consistently generated images preserve background regions, using masked PSNR and SSIM and visual comparison with CycleGAN.
- Metrics: Background consistency is evaluated with PSNR and SSIM between generated and original image backgrounds.The metrics target the requirement that object transformation should preserve the background.
- Metrics: Masked background PSNR compares the original and generated images after pixel-wise multiplication by the segmentation mask.The formulation uses the background portion represented by (1−m_i).
- Visual comparison: Figure 4 compares the input image, CycleGAN output, and Attention-GAN output for horse↔zebra transfiguration.The comparison is organized as three images per example.
- Evaluation data: MSCOCO test images and segmentation masks are used to evaluate generated-image background quality.The reported background-quality evaluation is conducted on the horse↔zebra task.
5.2 Unsupervised Results Comparisons to State-of-the-Art
Unsupervised Attention-GAN is compared with CycleGAN through qualitative examples, human preference tests, and background-consistency measures across object transfiguration tasks.
- Qualitative comparison: Attention-GAN produces more visually pleasing translations than CycleGAN on tiger↔leopard and apple↔orange examples.The comparison includes input, CycleGAN, and Attention-GAN results in Figure 5.
- Qualitative comparison: CycleGAN sometimes transforms background regions, whereas Attention-GAN preserves background consistency in the reported examples.Examples include changes to grass, trees, jeans, water, and weeds in CycleGAN outputs.
- Human evaluation: Human evaluation uses pairwise A/B tests on 100 randomly selected images per task, with 10 participants judging realism for each image.The tested tasks are horse↔zebra, tiger↔leopard, and apple↔orange.
- Quantitative comparison: Supervised Attention-GAN outperforms unsupervised Attention-GAN and CycleGAN on background PSNR and SSIM in the horse↔zebra task.The comparison is reported in Table 1.
- Quantitative comparison: For horse↔zebra, unsupervised Attention-GAN outperforms CycleGAN by a large margin on both background PSNR and SSIM.The evaluation uses MSCOCO test data and segmentation masks.
5.3 Model Analysis
Model analysis on horse-to-zebra translation shows that attention separates objects from backgrounds, while sparse-loss strength controls the balance between background consistency and object coverage.
- Generation analysis: Attention maps disentangle objects of interest from backgrounds, enabling transformation outputs to focus on the object while final images recombine transformed objects with backgrounds.Figure 7 presents inputs, attention maps, transformation outputs, factorized backgrounds and objects, and final composites.
- Sparse-loss ablation: Without sparse loss, the attention network incorrectly includes background regions among objects of interest.This corresponds to λattn = 0 in the ablation analysis.
- Sparse-loss ablation: Increasing λattn improves background consistency, but excessive sparsity shrinks the mask and reduces transformed-object quality.The analysis identifies λattn = 5 as excessive because the mask fails to cover the whole object.
- Sparse-loss ablation: λattn = 1 provides an appropriate balance between concentrating attention and comprehensively locating the object.The paper identifies this setting as a suitable choice for the horse-to-zebra task.
5.4 Comparisons of Supervised Results
Supervised attention improves object localization and background consistency relative to unsupervised Attention-GAN and CycleGAN. Qualitative comparisons show that supervision reduces erroneous transformations of background or non-target regions.
- Quantitative comparison: Supervised Attention-GAN outperforms unsupervised Attention-GAN and CycleGAN on background-region PSNR and SSIM across the horse-to-zebra test set.The comparison evaluates background consistency between generated and original images.
- Qualitative comparison: Supervised attention more clearly separates backgrounds from target objects and detects objects with higher confidence.The supervised attention maps tend to be dark red or dark blue.
- Qualitative comparison: CycleGAN and unsupervised Attention-GAN transform parts of human faces into black-and-white stripes in the comparison examples.The first row is given as a specific example of non-target regions being transformed.
6 Conclusion
The paper introduces attention into GAN-based object transfiguration by separating object localization from object transformation. Its results support attention modeling for producing high-quality, practically preferred transfigurations.
- Conclusion: The proposed three-player model uses separate attention, transformation, and discriminative networks for object transfiguration.The attention network predicts regions of interest, while the transformation network translates objects between domains.
- Conclusion: Attentional cycle consistency and sparse losses encourage attention to remain focused on objects of interest.These losses are introduced to guide the attention mechanism.
- Conclusion: Results demonstrate the necessity of investigating attention and show that the algorithm produces high-quality, practically preferred object transfigurations.